2018년 9월 10일 월요일

[Automate Build] 6. Fix My Build.xml (Apache Ant)


At the previous blog, I got error message ' "SomeDirectory/clz" not found'.
My build.xml is working well at Eclipse but, it is not enough to run in Jenkins.
<?xml version="1.0" encoding="UTF-8"?>
<project name="TestSample">

    <!-- set system properties -->
    <property name="home" value=".." />
    <property name="java6.home" value="C:/Program Files/Java/jdk1.6.0_45" />
    
    <!-- set proeject path properties -->
    <property name="project_01" value="${home}/project_01" />
    <property name="project_02" value="${home}/project_02" />
    
    <!-- set library path properties -->
    <property name="lib_01" value="${someProject}/lib/lib_01" />
    <property name="lib_01" value="${someProject}/lib/lib_02" />

    <!-- set build target -->
    <target name="Build All Group" depends="Group_01, Group_02" />
    <target name="Group_01" depends="build_project_01" />
    <target name="Group_02" depends="build_project_02" />

    <target name="build_project_01">
        <delete dir="${someProject}">
            <include name="project_01.jar" />
        </delete>
        <jar destfile="${someProject}/build_project_01.jar" basedir="${project_01}/clz" />
    </target>

    <target name="build_project_02">
        <delete dir="${someProject}">
            <include name="project_02.jar" />
        </delete>
        <jar destfile="${someProject}/build_project_02.jar" basedir="${project_02}/clz" />
    </target>

</project>


I will modify to compile with keeping original structure.
(This series' purpose is introduce auto-building system with little learning curve to my co-workers)

First, I need to make directory 'clz' instead of Eclipse.
<target name="build_project_01">

    <!-- add mkdir -->
    <delete dir="${project_01}/clz"/>
    <mkdir dir="${project_01}/clz"/>
</target>


Adding compile command.
<property name="java6.javac" location="${java6.home}/bin/javac"/>

<target name="build_project_01">
    
    <!-- add javac for compile -->
    <javac executable="${java6.javac}" fork="yes" includeantruntime="false" encoding="UTF-8"
            debug="true" debuglevel="lines,vars,source" srcdir="${project_01}/src" destdir="${project_01}/clz">
        <classpath>
            <fileset dir="${lib_01}" includes="*.jar"/>
            <fileset dir="${lib_02}" includes="*.jar"/>
        </classpath>
    </javac>

</target>


Copy files witout '.java' such as '.hbm', '.jpg' and '.js'
<target name="build_project_01">

    <!-- add copy command -->
    <copydir src="${project_01}/src" dest="${project_01}/clz" excludes="**/*.java"/>

</target>


Finally, change java.home to run build.xml in Jenkins.

<!-- java6.home path for using in eclipse -->
<!-- property name="java6.home" value="C:/Program Files/Java/jdk1.6.0_45" /-->

<!-- java6.home path for using in Jenkins -->
<property name="java6.home" value="${java.home}/.." />

<property name="java6.javac" location="${java6.home}/bin/javac"/>


This is full sample build.xml file in my case
Actual my build.xml is longer than this, but basically repeating this structure.
<?xml version="1.0" encoding="UTF-8"?>
<project name="TestSample">

    <!-- set system properties -->
    <property name="home" value=".." />
    <property name="java6.home" value="C:/Program Files/Java/jdk1.6.0_45" />
    
    <!-- set proeject path properties -->
    <property name="project_01" value="${home}/project_01" />
    <property name="project_02" value="${home}/project_02" />
    
    <!-- set library path properties -->
    <property name="lib_01" value="${someProject}/lib/lib_01" />
    <property name="lib_01" value="${someProject}/lib/lib_02" />

    <!-- set build target -->
    <target name="Build All Group" depends="Group_01, Group_02" />
    <target name="Group_01" depends="build_project_01" />
    <target name="Group_02" depends="build_project_02" />

    <target name="build_project_01">
        <delete dir="${project_01}/clz"/>
        <mkdir dir="${project_01}/clz"/>

        <javac executable="${java6.javac}" fork="yes" includeantruntime="false" encoding="UTF-8"
                debug="true" debuglevel="lines,vars,source" srcdir="${project_01}/src" destdir="${project_01}/clz">
            <classpath>
                <fileset dir="${lib_01}" includes="*.jar"/>
                <fileset dir="${lib_02}" includes="*.jar"/>
            </classpath>
        </javac>

        <copydir src="${project_01}/src" dest="${project_01}/clz" excludes="**/*.java"/>

        <delete dir="${someProject}">
            <include name="project_01.jar" />
        </delete>
        <jar destfile="${someProject}/build_project_01.jar" basedir="${project_01}/clz" />
    </target>

    <target name="build_project_02">
        <delete dir="${project_02}/clz"/>
        <mkdir dir="${project_02}/clz"/>

        <javac executable="${java6.javac}" fork="yes" includeantruntime="false" encoding="UTF-8"
                debug="true" debuglevel="lines,vars,source" srcdir="${project_02}/src" destdir="${project_02}/clz">
            <classpath>
                <fileset dir="${lib_01}" includes="*.jar"/>
                <fileset dir="${lib_02}" includes="*.jar"/>
            </classpath>
        </javac>

        <copydir src="${project_02}/src" dest="${project_02}/clz" excludes="**/*.java"/>

        <delete dir="${someProject}">
            <include name="project_02.jar" />
        </delete>
        <jar destfile="${someProject}/build_project_02.jar" basedir="${project_02}/clz" />
    </target>

</project>


Set modified build.xml (build_in_jenkins.xml)

It is result running build.xml

Success!!

2018년 8월 7일 화요일

[Automate Build] 5. Create New Item in Jenkins

5. Create New Item in Jenkins

Let's create new item


I don't know well about Jenkins, so I choose 'Freestyle project'.


Fill in some words on blanks.

I want to integrate with SVN server. I must make credentials about svn user


Fill in some words on blanks too.


My company's build tools is Ant, specifically Ant 1.7.0 version and It based on JDK 1.6.0_45


At build tab, I use invoke Ant. Fill in proper values and save


I build this Item manually


It has an error. Go to console output


My build.xml file has a problem. I will fix Build.xml file next.
* I blind somethig that can have problem, but it's not important things to jenkins

2018년 7월 27일 금요일

[Automate Build] 4. Setup Configurations in Jenkins

4. Setup Configurations in Jenkins

I need to setup configurations.
My company's tool has been built by Jdk1.6.0_45 and ant1.7.0.
(I don't like these old versions. However, I don't have authority changing this.)

Go to 'Configuration System'


Select SVN version and save


Go to 'Global Tool Configuration'


Fill in the name and select version.
I select jdk1.6.0_45. You can select other proper version for you
For download jdk from oracle click this


Fill in your oracle account informations


Saved




Fill in the name and select version
I select and1.7.0

2018년 7월 22일 일요일

[Automate Build] 3. Getting Started Jenkins

3. Getting Started Jenkins

Input initial password

I don't know well about Jenkins, so I choose 'Install sugested plugins'

Some plugins are installing

I need to make first admin user

Jenkins is ready!

[Automate Build] 2. Set up Jenkins Server on Docker

2. Set up jenkins server on docker
# mkdir -p /share/docker/jenkins
# docker run -d -p 8080:8080 -v /share/docker/jenkins:/var/jenkins_home --name jenkins -u root jenkins
------ update 2018-02-26 ------
** error : I got an error. when I installed jdk 1.7 on Jenkins on the virtualBox SharedFolder(/share/).

# mkdir -p /docker/jenkins
# docker run -d -p 8080:8080 -v /docker/jenkins:/var/jenkins_home --name jenkins -u root jenkins

# docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED              STATUS                   PORTS                               NAMES
c78da9c5ae57        jenkins                    "/bin/tini -- /usr/l…"   About a minute ago   Up About a minute        0.0.0.0:8080->8080/tcp, 50000/tcp   jenkins
a95a445b29c8        garethflowers/svn-server   "/usr/bin/svnserve -…"   14 hours ago         Up 4 minutes (healthy)   0.0.0.0:3690->3690/tcp              svn-server

Access jenkins page through browser and you need initial password
# docker exec -it jenkins cat /var/jenkins_home/secrets/initialAdminPassword
a5a62c8d573c4a5682993f0820680349

2-2.Optional
I want jenkins' workspace to be on the virtualBox SharedFolder
Make new virtualBox SharedFolder in setting.


Mount jenkins-workspace
# sudo mount -t vboxsf jenkins_workspace /docker/jenkins/workspace/

2018년 4월 29일 일요일

[Automate Build] 1. Set up SVN Server on Docker

1. Set up SVN Server on Docker
I set up svn server on docker
(SVN server in my company is on the macOS, but it's not matter.)
Install svn server and make repository
# mkdir -p /share/docker/svn-server/svn-root

# docker run --name svn-server \
           --detach \
           --volume /share/docker/svn-server/svn-root:/var/opt/svn \
           --publish 3690:3690 \
           garethflowers/svn-server

# docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS                           PORTS                    NAMES
a95a445b29c8        garethflowers/svn-server   "/usr/bin/svnserve -…"   3 seconds ago       Up 1 second (health: starting)   0.0.0.0:3690->3690/tcp   svn-server

# docker exec -it svn-server svnadmin create test-repo

# docker exec -it svn-server svnadmin info test-repo
Path: test-repo
UUID: 16f1dd00-4a5d-44f2-adcb-24c03d1b441e
Repository Format: 5
Compatible With Version: 1.9.0
Repository Capability: mergeinfo
Filesystem Type: fsfs
Filesystem Format: 7
FSFS Sharded: yes
FSFS Shard Size: 1000
FSFS Shards Packed: 0/0
FSFS Logical Addressing: yes
Configuration File: test-repo/db/fsfs.conf

Go to container's CLI to set config or add users
# docker exec -it svn-server sh

1-2. Add svn users
# vi /var/opt/svn/test-repo/conf/passwd
### This file is an example password file for svnserve.
...
[users]
# harry = harryssecret
# sally = sallyssecret
seunghyun = 1234567
* Caution : hot key 'shift+Q' doesn't work, use ':' + 'wq'

# vi /var/opt/svn/test-repo/conf/authz
### This file is an example authorization file for svnserve.
...
[groups]
# harry_and_sally = harry,sally
# harry_sally_and_joe = harry,sally,&joe
default = seunghyun

# [/foo/bar]
# harry = rw
# &joe = r
# * =

[/]
@default = rw

...
* Caution : hot key 'shift+Q' doesn't work, use ':' + 'wq'

# vi /var/opt/svn/test-repo/conf/svnserve.conf
### This file controls the configuration of the svnserve daemon, if you
...
[general]
anon-access = none
auth-access = write
password-db = passwd
authz-db = authz
* Caution : hot key 'shift+Q' doesn't work, use ':' + 'wq'

2018년 2월 17일 토요일

[Automate Build] 0. Introduce - current system state

I am in charge of QA  in small Data Integration Tool company since I began to work in Jan 2017. QA team's role of our company is something special. QA Team must test product's functions, performance and UI, also fix bugs, keep VCS(svn) and report product's improvements. Sometimes, we develop new function and guide operation team how to use function property.

We have two members in our QA team include me, but senior and I don't have experience about this QA role. Therefore, we had to learn about QA role ourselves. I learned product's functions and working process on qa team during the first year. In second year, I think I should introduce CD and CI if I can.

These are my tasks.
1. Test product (function, performance, UI)
2. Fix bugs
3. Develop new functions
4. Build & Release product
5. Keep VCS(subversion)


1. Test product
There is not specific test. We have only one test checklist that is thirty pages include UI, function and performance to release. It complicated three type of test. There is not unit test. We have tested everything manually. I want to make test automation. Whenever svn was committed, I have to test A to Z manually. It takes so many times.

2. Fix bugs
When bug is reported to QA team, team manager want to fix bugs in QA team.
When bug is come from new source code, team manager also want to fix bugs in QA team. I think bug reports go to man who developed this function is better than come to qa team. I can accept this. But, I want to focus on QA and make CD, CI

3. Support to create new function
Customer requires new functions. Sometimes, I find technique for new functions. I make sample code and explain to software engineer about technique that I found.

4. Build & Release product
Product build manually by Ant at eclipse. This has problems. People must make mistake. If eclipse's environments was changed to do something, building will be wrong. It occurred many time for a year. It must change to automate

5. Keep subversion
We use only trunk branch in svn. It's terrible. All codes didn't test enough to release but, they are committed to trunk. We must use branches. I will make svn's branch strategy and guild this.

[Current work flow]
1. Test



2. Commit Source Code




Most needs
1. Function Test Automation.
2. Performance Test Automation.
3. UI Test Automation.
4. Integration Test Automation.
These spend time a lot. Especially, function test is very important last gate to deploy.


Future Tools
Version Control Tool : SVN
Continuous Integration Server : Jenkins
Issue Tracking System : redmine
Build Tool : ant (hope to use gradle)
UnitTest : JUnit
UITestTool : Katalon Studio & selenium
FunctionTestTool : SoapUI
Code Coverage Tool : Not yet.


Docker run command simple reference

docker run
Run command create and start container

If you don't have image to make container, Docker automatically search proper image and pull image.

1. you don't have image
# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

# docker run ubuntu
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
1be7f2b886e8: Pull complete 
6fbc4a21b806: Pull complete 
c71a6f8e1378: Pull complete 
4be3072e5a37: Pull complete 
06c6d2f59700: Pull complete 
Digest: sha256:e27e9d7f7f28d67aa9e2d7540bdc2b33254b452ee8e60f388875e5b7d9b2b696
Status: Downloaded newer image for ubuntu:latest

# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
d23a16d47ae7        ubuntu              "/bin/bash"         18 seconds ago      Exited (0) 17 seconds ago                       stoic_johnson

# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              latest              0458a4468cbc        3 weeks ago         112MB

2. you have image
# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu              latest              0458a4468cbc        3 weeks ago         112MB

# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

# docker run ubuntu

# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
6fe3a86ac19a        ubuntu              "/bin/bash"         9 seconds ago       Exited (0) 8 seconds ago                       peaceful_kare

[options]

-it
Using for interactive processes
# docker run -it ubuntu:latest

root@b9f52826da9c:/# ll
total 72
drwxr-xr-x   1 root root 4096 Feb 17 03:58 ./
drwxr-xr-x   1 root root 4096 Feb 17 03:58 ../
-rwxr-xr-x   1 root root    0 Feb 17 03:58 .dockerenv*
drwxr-xr-x   2 root root 4096 Jan 23 22:49 bin/
drwxr-xr-x   2 root root 4096 Apr 12  2016 boot/
drwxr-xr-x   5 root root  360 Feb 17 03:58 dev/
drwxr-xr-x   1 root root 4096 Feb 17 03:58 etc/
drwxr-xr-x   2 root root 4096 Apr 12  2016 home/
drwxr-xr-x   8 root root 4096 Sep 13  2015 lib/
drwxr-xr-x   2 root root 4096 Jan 23 22:49 lib64/
drwxr-xr-x   2 root root 4096 Jan 23 22:49 media/
drwxr-xr-x   2 root root 4096 Jan 23 22:49 mnt/
drwxr-xr-x   2 root root 4096 Jan 23 22:49 opt/
dr-xr-xr-x 118 root root    0 Feb 17 03:58 proc/
drwx------   2 root root 4096 Jan 23 22:49 root/
drwxr-xr-x   1 root root 4096 Jan 23 22:49 run/
drwxr-xr-x   1 root root 4096 Jan 25 18:23 sbin/
drwxr-xr-x   2 root root 4096 Jan 23 22:49 srv/
dr-xr-xr-x  13 root root    0 Feb 17 03:58 sys/
drwxrwxrwt   2 root root 4096 Jan 23 22:49 tmp/
drwxr-xr-x   1 root root 4096 Jan 23 22:49 usr/
drwxr-xr-x   1 root root 4096 Jan 23 22:49 var/

-d
To start a container in detached mode (Running in background)
* Caution : Container will stop when any foreground process don't work in container
I use option '-it' to show running container detached mode on below code
# docker run -it -d ubuntu:latest 
8bb0beb95f0f9f85cbf781650baa2194693c6eb5a425a30bcd5c03b61b1539f3

# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
8bb0beb95f0f        ubuntu:latest       "/bin/bash"         4 seconds ago       Up 4 seconds                            quizzical_goodall

-p
Publish a container᾿s port or a range of ports to the host
docker run -p hostport:containerport
# docker run -d -p 12345:6379 redis
efb01ae0a176e72a1e0c7c1e0b418e1e8d338bf2366c2991af12ad8712c32933

$ telnet localhost 12345
Trying ::1...
Connected to localhost.
Escape character is '^]'.
set test abcde
+OK
get test
$5
abcde
^]
telnet> quit
Connection closed.

--rm
automatically clean up the container and remove the file system when the container exits
seunghyun@ubuntu:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
seunghyun@ubuntu:~$ docker run -it ubuntu
root@19c1e015546a:/# exit
exit
seunghyun@ubuntu:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
19c1e015546a        ubuntu              "/bin/bash"         6 seconds ago       Exited (0) 3 seconds ago                       quirky_booth
seunghyun@ubuntu:~$ docker run -it --rm ubuntu
root@28e77f58194b:/# exit
exit
seunghyun@ubuntu:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
19c1e015546a        ubuntu              "/bin/bash"         27 seconds ago      Exited (0) 24 seconds ago                       quirky_booth

--name
Defining a name can be a handy way to add meaning to a container
seunghyun@ubuntu:~$ docker run --name seunghyun ubuntu
seunghyun@ubuntu:~$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
seunghyun@ubuntu:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
ee741713119e        ubuntu              "/bin/bash"         5 seconds ago       Exited (0) 4 seconds ago                       seunghyun

reference : https://docs.docker.com/engine/reference/run/

2018년 2월 14일 수요일

Mounting Virtual Box sharedFolder on Ubuntu 16.04

1. Install VBoxGuestAdditions.iso
# apt-get install virtualbox-guest-additions-iso
# mkdir -p /media/iso
# mount /usr/share/virtualbox/VBoxGuestAdditions.iso /media/iso/
# sh /media/iso/VBoxLinuxAdditions.run

2. set sharedFolder on VirtualBox
VirtualBox - Setting - Shared Folders - adds new shared folder

In this case, I set folder named 'hostFolder' on hostOS


3. manual mount
# mkdir /share
# mount -t vboxsf hostFolder /share

4. set auto mount on boot
You should uncheck Auto-mount option in VirtualBox's setting.
# vi /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
...
/dev/mapper/ubuntu--vg-swap_1 none            swap    sw              0       0

# append below code
# mount sharedFolder on boot
hostFolder /share vboxsf defaults 0 0

# vi /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

# append below code
# mount sharedFolder on boot
vboxsf

# reboot

2018년 2월 5일 월요일

Host file copy to container, Container file copy to host

File copy

1. Host to Container
$ docker cp /path/foo.bar containername:/path/foo.bar

2. Container to Host
$ docker cp containername:/path/foo.bar /path/foo.bar