Create self signed certificate in ubuntu

sudo mkdir /etc/apache2/ssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/apache2/ssl/apache.key -out /etc/apache2/ssl/apache.crt

Tweaks to make puppet passenger working

I have been writing a puppet script to create VMs with ruby, rails and passenger recently and encountered quite a few small but annoying problems. Just want to write it down since it may help others.

Environments:

    Ubuntu 14

    Ruby 2.1.7

    Rails 4.2.5

    Passenger 5.0.22

    Puppet 3.4.3

Puppet modules:

    puppetlabs-concat

    puppetlabs-apache

    puppetlabs-passenger

    puppetlabs-apt

    puppetlabs-vcsrepo

1. Problem with puppetlabs/ruby v0.4.0

This module does not support Ruby version 2.2.x, so have to use Ruby '2.1.0'.

2. Problems with puppetlabs/passenger v0.4.1

2.1) Passenger version: to specify the version of passenger, according to the documentation and the example in the README, only passenger_version needs to be set. However, it is NOT working at all. Both passenger_version and package_ensure need to be set, e.g.

class {'passenger':

passenger_version      => "$passenger_version", 

package_ensure          => "$passenger_version", 

passenger_provider     => 'gem',

......

  }  

2.2) RailsAutoDetect: Passenger module adds INVALID 'RailsAutoDetect' into /etc/apache2/mods-enabled/passenger.conf. To fix it:

  file_line { 'Change RailsAutoDetect to PassengerEnabled':

 path => '/etc/apache2/mods-enabled/passenger.conf',  

 line => '        PassengerEnabled On',

 match   => "RailsAutoDetect",

 ......

  }
Update: the puppet passenger module master branch has fixed this bug but has not published to forge as of Jan 2016.

2.3) HTTP 403 error on apps: Passenger module adds 'PassengerHighPerformance On' into /etc/apache2/mods-enabled/passenger.conf. It gives an HTTP 403 error to webapps sitting on a sub path. To fix it:

  file_line { 'Disable PassengerHighPerformance':

 path => '/etc/apache2/mods-enabled/passenger.conf',  

 line => '        PassengerHighPerformance off',

 match   => "PassengerHighPerformance",

 ......

  }
Update: a better choice is to set PassengerHighPerformance off in the <Location xxx> directive.

2.4) Permissions: as the webapp will be running under apache user, you may want to add read permission to 'www-data' group to your webapp path.

3. Copy file from host to VM

Remember to remove carriage returns, for instance:

  file { 'post_config.sh':

    ensure => 'file',

source => "puppet:///modules/my_module/post_config.sh",

path   => '/app_path/post_config.sh',

        ,,,,,,

before  => Exec['Remove_carriage_returns'],

  }

  

  exec { 'Remove_carriage_returns':

    command  => "/bin/sed -i s/\r//g /app_path/post_config.sh",

require  => File['/app_path/post_config.sh'],

  }

4. Cannot create user with password
If you want to use 'user' to create a user with password, by default it WONT work, e.g. the password in /etc/shadow will be EMPTY!
  user { 'user1' :
    ensure   =>  'present',
password =>  'SHA512_PASSWORD'
shell       =>  '/bin/bash',
home     =>  '/home/user1',
managehome => true,
  }
The reason is that a ruby gem 'ruby-shadow' must be installed first! (Thanks to puppet cookbook)
package { 'ruby-shadow':
    ensure   => 'present',
    provider => 'gem',
}

Working example on OAuth2 + Spring Security + Apache Oltu

Components:
  Spring MVC + Security  3.0.5.RELEASE
  Apache Oltu 1.0.1-SNAPSHOT

POM:
       <!-- add spring / security / mvc dependencies -->

        <!--DO NOT USE version 1.0.0 which causes oauth2 phrase2 auth problem with Google -->
        <dependency>
            <groupId>org.apache.oltu.oauth2</groupId>
            <artifactId>org.apache.oltu.oauth2.client</artifactId>
            <version>1.0.1-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.apache.oltu.oauth2</groupId>
            <artifactId>org.apache.oltu.oauth2.common</artifactId>
            <version>1.0.1-SNAPSHOT</version>
        </dependency>

Step 1: send request to get authorization code

Apache Oltu has defined some oauth providers, e.g. google, facebook, then you need to do something like:
OAuthClientRequest request = OAuthClientRequest
                    .authorizationProvider(OAuthProviderType.GOOGLE)
                    .setClientId(YOUR_CLIENT_ID)
                    .setRedirectURI(YOUR_REDIRECT_URI)
                    .setResponseType("code")
                    .setScope("https://www.googleapis.com/auth/userinfo.email")
                    .buildQueryMessage();
response.sendRedirect(request.getLocationUri());

If the oauth provider is not defined in Oltu, then set authorization location manually:
OAuthClientRequest request = OAuthClientRequest
                        .authorizationLocation("https://api.login.yahoo.com/oauth2/request_auth")
                       .setClientId(YOUR_CLIENT_ID)
                       .setRedirectURI(YOUR_REDIRECT_URI)                        
                       .setResponseType("code")
                       .buildQueryMessage();
response.sendRedirect(request.getLocationUri());

For oauth providers not lised in Oltu, here are some URLs
YAHOO_AUTHORIZATION_URL = "https://api.login.yahoo.com/oauth2/request_auth";
YAHOO_TOKEN_URL  = "https://api.login.yahoo.com/oauth2/get_token";
YAHOO_GUID_URL     = "http://social.yahooapis.com/v1/me/guid";
YAHOO_USER_PROFILE_URL = "https://social.yahooapis.com/v1/user/__USERID__/profile?format=json";

TWITTER_REQUEST_TOKEN_URL = "https://api.twitter.com/oauth/request_token";
TWITTER_AUTHORIZATION_URL = "https://api.twitter.com/oauth/authorize";
TWITTER_TOKEN_URL         = "https://api.twitter.com/oauth/access_token";
TWITTER_PROFILE_URL       = "https://api.twitter.com/1.1/users/profile_banner.json"

Step 2: get access code

Do the following in your code linking to REDIRECT_URI

2.1 Get authorization code:

OAuthAuthzResponse oar = OAuthAuthzResponse.oauthCodeAuthzResponse(httpRequest);

String code = oar.getCode();

2.2 Use Oltu to get access token, e.g. from Google

OAuthClientRequest oauthRequest = OAuthClientRequest
                .tokenProvider(OAuthProviderType.GOOGLE)
                .setClientId(CLIENT_ID)
                .setClientSecret(CLIENT_SECRET)
                .setRedirectURI(REDIRECT_URI)
                .setGrantType(GrantType.AUTHORIZATION_CODE)
                .setCode(code)
                .buildBodyMessage();
OAuthClient oAuthClient = new OAuthClient(new URLConnectionClient());
final OAuthAccessTokenResponse oAuthResponse = oAuthClient.accessToken(oauthRequest, "POST");
String accessToken = oAuthResponse.getAccessToken();
long expiresIn    = oAuthResponse.getExpiresIn();

Same code to get access token from FACEBOOK

2.3 Use Apache Httpclient to get access token from Yahoo

I cannot get Oltu working with Yahoo due to the data being sent from Oltu (HTTP header) is not accepted by Yahoo

HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost     = new HttpPost(OAuth2Manager.YAHOO_TOKEN_URL);

List<NameValuePair> params = new ArrayList<NameValuePair>(2);
params.add(new BasicNameValuePair("client_id", CLIENT_ID);
params.add(new BasicNameValuePair("client_secret", CLIENT_SECRET);
params.add(new BasicNameValuePair("redirect_uri", REDIRECT_URI);
params.add(new BasicNameValuePair("code", code));
params.add(new BasicNameValuePair("grant_type", "authorization_code"));
params.add(new BasicNameValuePair("response_type", "code"));
httppost.setEntity(new UrlEncodedFormEntity(params, "UTF-8"));

String encodedValue=new String(Base64.encodeBase64(CLIENT_ID+":"+CLIENT_SECRET).getBytes("UTF-8"));
httppost.setHeader("Authorization", "Basic "+encodedValue);
HttpResponse oresponse = httpclient.execute(httppost);

org.apache.http.HttpEntity entity = oresponse.getEntity();

String theString = IOUtils.toString(entity.getContent(), "UTF-8");
// parse JSON string here ...
final JSONObject obj = new JSONObject(theString);
String access_token = obj.getString("access_token");
String xoauth_yahoo_guid = obj.getString("xoauth_yahoo_guid");

Step 3: retrieve user information

3.1 Google

OAuthClientRequest bearerClientRequest = new OAuthBearerClientRequest("https://www.googleapis.com/plus/v1/people/me")
                .setAccessToken(accessToken).buildQueryMessage();
OAuthResourceResponse resourceResponse = oAuthClient.resource(bearerClientRequest, "GET", OAuthResourceResponse.class);
String userinfoJSON = resourceResponse.getBody();
final JSONObject obj    = new JSONObject(userinfoJSON);
final JSONArray  emails = obj.getJSONArray("emails");
final int n = emails.length();
for (int i = 0; i < n; ++i) {
  final JSONObject email = emails.getJSONObject(i);
  String type = email.getString("type");
  if("account".equals(type)) {
      System.out.println(email.getString("value"));
  }
}
String googleid = obj.getString("id");
String fname     = obj.getJSONObject("name").getString("givenName");
String lname     = obj.getJSONObject("name").getString("familyName");

3.2 Facebook

A bit different from 3.1,

// Facebook’s response is not fully compliant with the final version of the OAuth 2 specification, but it can be parsed using the class GitHubTokenResponse.
GitHubTokenResponse oAuthResponse = oAuthClient.accessToken(oauthRequest, GitHubTokenResponse.class);
String accessToken = oAuthResponse.getAccessToken();

OAuthClientRequest bearerClientRequest = new OAuthBearerClientRequest("https://graph.facebook.com/me?fields=id,first_name,last_name,email")
                .setAccessToken(accessToken).buildQueryMessage();
OAuthResourceResponse resourceResponse = oAuthClient.resource(bearerClientRequest, "GET", OAuthResourceResponse.class);
String userinfoJSON = resourceResponse.getBody();
final JSONObject obj    = new JSONObject(userinfoJSON);

3.3 Yahoo

HttpGet getrequest = new HttpGet("https://social.yahooapis.com/v1/user/__USERID__/profile?format=json".replace("__USERID__", xoauth_yahoo_guid));
getrequest.addHeader("Authorization", "Bearer " + access_token);
HttpResponse getresponse = httpclient.execute(getrequest);
String userinfoJSON = EntityUtils.toString(getresponse.getEntity());

final JSONObject profileobj = new JSONObject(userinfoJSON);
String yahooid = profileobj.getJSONObject("profile").getString("guid")

Step 4: automatic login in Spring Security

public static void autoLogin(
                    AuthenticationProvider authenticationProvider,
                    String oauthProvider,
                    OAuth2User oauth2User,
                    HttpServletRequest request) {
        try {
            Set<GrantedAuthority> grantedAuthorities = new HashSet<GrantedAuthority>();
            grantedAuthorities.add(new GrantedAuthorityImpl("ROLE1"));
            grantedAuthorities.add(new GrantedAuthorityImpl("ROLE2"));

            // check if user is admin
            if (adminIDs.contains(oauth2User.getID())) {
                grantedAuthorities.add(new GrantedAuthorityImpl("admin"));
            }

            UsernamePasswordAuthenticationToken token = new UsernamePasswordAuthenticationToken("USER", "PASS", grantedAuthorities);
            token.setDetails(new WebAuthenticationDetails(request));
            Authentication authentication = authenticationProvider.authenticate(token);

            SecurityContextHolder.getContext().setAuthentication(authentication);
         
            // Create a new session and add the security context.
            HttpSession session = request.getSession(true);
            session.setAttribute("SPRING_SECURITY_CONTEXT", SecurityContextHolder.getContext());
        } catch (Exception e) {
            SecurityContextHolder.getContext().setAuthentication(null);
            logger.error("Failure in autoLogin", e);
        }
    }

How to get Solr to log to a log file

This post is just a summary of somebody's post in here: http://skybert.wordpress.com/2009/07/22/how-to-get-solr-to-log-to-a-log-file/


1. Edit $TOMCAT_HOME/conf/logging.properties

# added 6localhost.org.apache.juli.FileHandler to the list of handlers

6localhost.org.apache.juli.FileHandler.level = FINE
6localhost.org.apache.juli.FileHandler.directory = /var/log/tomcat6/solr
6localhost.org.apache.juli.FileHandler.prefix = solr.

org.apache.solr.level=INFO
org.apache.solr.handlers=6localhost.org.apache.juli.FileHandler



2. make sure u put the handler in the list of handlers in the line,

handlers = 1catalina.org.apache.juli.FileHandler, ……


3. also make sure ur log output directory is right..
6localhost.org.apache.juli.FileHandler.directory = /var/log/tomcat6/solr

Thickbox does not show scrollbar in ipad and solution

When I use thickbox to popup an iframe in my page, it shows in ipad WITHOUG scrollbar. Solution: 1) Edit thickbox.js Find tb_init(), add // identify the touch device if ("ontouchstart" in document.documentElement){ $('html').addClass('touch'); } 2) Edit thickbox.css Add: .touch #TB_window { height: 90%; -webkit-overflow-scrolling: touch !important; overflow: scroll !important; }

Generate a screenshot in Java

Under ubuntu,

1. Install firefox and virtual framebuffer

sudo aptitude install xvfb firefox

2. Write Java code:
import java.io.File;
import java.io.IOException;

import org.apache.commons.io.FileUtils;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxBinary;
import org.openqa.selenium.firefox.FirefoxDriver;

public class CaptureScreenshotTest{
   private static int DISPLAY_NUMBER = 99;
   private static String XVFB = "/usr/bin/Xvfb";
   private static String XVFB_COMMAND = XVFB + " :" + DISPLAY_NUMBER;
   private static String URL = "http://www.google.com/";
   private static String RESULT_FILENAME = "/tmp/screenshot.png";

   public static void main ( String[] args ) throws IOException{
     Process p = Runtime.getRuntime().exec(XVFB_COMMAND);
     FirefoxBinary firefox = new FirefoxBinary();
     firefox.setEnvironmentProperty("DISPLAY", ":" + DISPLAY_NUMBER);
     WebDriver driver = new FirefoxDriver(firefox, null);
     driver.get(URL);
     File scrFile = ( (TakesScreenshot) driver ).getScreenshotAs   (OutputType.FILE);
     FileUtils.copyFile(scrFile, new File(RESULT_FILENAME));
     driver.close();
     p.destroy();
   }
}


3. Get the following jars:
selenium-java-2.37.0.jar
commons-io-2.4.jar
guava-15.0.jar
json-jena-1.0.jar
commons-exec-1.1.jar
httpcore-4.3.jar
httpclient-4.3.jar
commons-logging-1.1.1.jar

4. Compile
javac -cp /jars/selenium-java-2.37.0.jar:/jars/commons-io-2.4.jar:. CaptureScreenshotTest.java


5. Run
java -cp /jars/selenium-java-2.37.0.jar:/jars/commons-io-2.4.jar:/jars/guava-15.0.jar:/jars/json-jena-1.0.jar:/jars/commons-exec-1.1.jar:/jars/httpcore-4.3.jar:/jars/httpclient-4.3.jar:/jars/commons-logging-1.1.1.jar:. CaptureScreenshotTest

The problem with Process.waitFor() method

I have recently experienced a strange problem while calling Lame to do a music file conversion. For some audio files, my application just hangs without any output.The code is shown as follows:

Runtime rt = Runtime.getRuntime();
Process p = rt.exec(commands);
p.waitFor();
int exitVal = p.exitValue();

Eventually I found that problem was caused by a process dead lock. In short, Lame produces lots output to stderr/stdout but the buffer size is fixed, therefore, Lame expects my application to consume/clear the buffer before it can continue processing. However, when I called p.waitFor(), my application is expecting Lame to exit. In such a situation, a dead lock is happending! To fixed the problem:

Runtime rt = Runtime.getRuntime();
Process p = rt.exec(commands);
InputStream stderr = p.getErrorStream();
InputStreamReader isr = new InputStreamReader(stderr);
BufferedReader br = new BufferedReader(isr);
String line = null;
while ( (line = br.readLine()) != null)
    logger.info(line);

int exitVal = p.waitFor();

For more information, please see this post, I know it is a post in 2000, but it still applies!

http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html?page=1