Timekeeper is a Java/Groovy library that helps tests to author performance reports in Markdown.
Often I develop Web UI tests in Groovy using Selenium. I want to measure the performance of the tests. Often I want to measure:
how long (seconds) tests take to navigate to a URL in browser
how long tests take to take and save screenshot of a web page
how large (bytes) is the generated image file
And I want to examine many URLs; 100 or more. In practice, most of URL respond within 10 seconds but a few of them sometimes respond slow (over 30 seconds). Why? What happened? I need to list the slow URLs and look into them.
The 1st problem is that it is bothersome recording the duration using a stopwatch device. I introduced the Apache Commons StopWatch library into my test scripts to measure the duration and print the figure in console messages.
The 2nd problem is that it is difficult to find useful information out of the bulk of console messages. I want to summarise the statistics. But it is too tiresome to write manually a statistics report in Markdown table format.
I want to automate these tasks entirely. I want my tests to perform not only measuring performance but also compiling a concise report in Markdown format. Here comes the Timekeeper!
Here is an outline of a test script which uses the Timekeeper to measure performance and print a report.
Your test script should create a Measurement
object, which is a container of Record
objects. A Measurement
requires you to define a set of table column names, like “Case” ad “URL”.
While performing a test (e.g, visiting URLs), your test script will make a record, and write the “startAt” timestamp before an action, and the “endAt” timestamp after the action. You may also put some “size” information into the record (e.g, the size of downloaded file).
The Record
objects should be stored in the Measurement
object.
Your test script will repeat creating Record
s and putting them into the Measurement
object as many times as you wants to. For example, you may visit 100 URLs and create 100 Records in a Measurement.
Your test script wants to make a Table
object, which wraps a Measurement
object and the information how you want it formatted in a text report. For example, you can specify how the rows of the Table to be sorted.
Your test script want to create a Timekeeper
object. Your script will put one or more Table
objects. Finally your test will call Timekeeper’s report() method which will write a text file. The report will contain one or more tables in Markdown format.
The following example is a minimalistic example of utilizing the Timekeeper library, in Groovy using JUnit5.
package com.kazurayam.timekeeper_demo
import com.kazurayam.timekeeper.Measurement
import com.kazurayam.timekeeper.ReportOptions
import com.kazurayam.timekeeper.Table
import com.kazurayam.timekeeper.Timekeeper
import org.junit.jupiter.api.BeforeAll
import org.junit.jupiter.api.Disabled
import org.junit.jupiter.api.Test
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.Paths
// This test takes 1 minutes to finish
@Disabled
class TimekeeperDemoMinimal {
private static Path outDir_
@BeforeAll
static void setupClass() {
outDir_ = Paths.get(".")
.resolve("build/tmp/testOutput")
.resolve(TimekeeperDemoMinimal.class.getSimpleName())
if (Files.exists(outDir_)) {
outDir_.toFile().deleteDir();
}
Files.createDirectory(outDir_)
}
@Test
void demo_planned_sleep() {
Measurement m1 =
new Measurement.Builder("How long it waited", ["Case"])
.build()
doRecording(m1)
Timekeeper tk = new Timekeeper.Builder()
.table(new Table.Builder(m1).build())
.build()
tk.report(outDir_.resolve("planned_sleep.md"))
}
private static void doRecording(Measurement m1) {
for (int i in [2, 3, 1]) {
m1.before(["Case": "sleeping for " + i + " secs"])
Thread.sleep(i * 1000L) // do something that takes long time.
m1.after()
}
}
This code outputs the following Markdown text.
## How long it waited
as events flowed
|Case|duration|graph|
|:----|----:|:----|
|sleeping for 13 secs|00:13|`##`|
|sleeping for 3 secs|00:03|`#`|
|sleeping for 7 secs|00:07|`#`|
|Average|00:07| |
The format of duration is "minutes:seconds"
one # represents 10 seconds in the duration graph
This Markdown text will be rendered like this:
The following code processes a list URLs. It makes HTTP GET request, save the request body into file. It checks the size of the file in bytes, and measures the duration of HTTP GET request.
package com.kazurayam.timekeeper_demo
import com.kazurayam.timekeeper.Measurement
import com.kazurayam.timekeeper.ReportOptions
import com.kazurayam.timekeeper.RowOrder
import com.kazurayam.timekeeper.Table
import com.kazurayam.timekeeper.Timekeeper
import org.junit.jupiter.api.BeforeAll
import org.junit.jupiter.api.Disabled
import org.junit.jupiter.api.Test
import java.nio.file.Files
import java.nio.file.Path
import java.nio.file.Paths
import java.time.LocalDateTime
// this test takes 3 minutes to finish
@Disabled
class TimekeeperDemoHttpInteraction {
private static Path outDir_
private static List<String> urlList = [
"case 1|https://www.google.com/search?q=timekeeper",
"case 1|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web",
"case 1|https://search.yahoo.co.jp/search?p=timekeeper",
"case 2|https://www.google.com/search?q=timekeeper",
"case 2|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web",
"case 3|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web",
]
@BeforeAll
static void setupClass() {
outDir_ = Paths.get(".")
.resolve("build/tmp/testOutput")
.resolve(TimekeeperDemoHttpInteraction.class.getSimpleName())
if (Files.exists(outDir_)) {
outDir_.toFile().deleteDir()
}
Files.createDirectory(outDir_)
}
@Test
void test_HTTPGetAndSaveResponse() {
Measurement interactions = new Measurement.Builder(
"get URL, save HTML into file", ["Case", "URL"])
.build()
// interact with URL, save the HTML into files
processURLs(urlList, outDir_, interactions)
// print the report
Timekeeper tk = new Timekeeper.Builder()
.table(new Table.Builder(interactions).build())
.build();
tk.report(outDir_.resolve("report.md"))
}
static void processURLs(List<String> urlList, Path outDir, Measurement m) {
for (line in urlList) {
String[] items = line.split("\\|")
URL url = new URL(items[1])
// mark the startAt timestamp
m.before(["Case": items[0], "URL": items[1]])
// do a heavy task
String content = getHttpResponseContent(url)
LocalDateTime afterGet = LocalDateTime.now()
File outFile = outDir.resolve(url.getHost() + ".html").toFile()
outFile.text = content
// record the file size and how long it took to finish the process
m.after(outFile.length())
}
}
static String getHttpResponseContent(URL url) {
HttpURLConnection con = (HttpURLConnection) url.openConnection()
con.setRequestMethod("GET")
con.setConnectTimeout(5000)
con.setReadTimeout(5000)
int status = con.getResponseCode()
Reader streamReader
if (status > 299) {
streamReader = new InputStreamReader(con.getErrorStream())
} else {
streamReader = new InputStreamReader(con.getInputStream())
}
BufferedReader br = new BufferedReader(streamReader)
String inputLine
StringBuffer sb = new StringBuffer()
while ((inputLine = br.readLine()) != null) {
sb.append(inputLine)
}
streamReader.close()
con.disconnect()
// sleep a while in between 1.0 - 5.0 seconds at random
Thread.sleep((long)(4000 * Math.random() + 1000))
return sb.toString()
}
This code will output the following Markdown text.
## get URL, save HTML into file
as events flowed
|Case|URL|size|duration|graph|
|:----|:----|----:|----:|:----|
|case 1|https://www.google.com/search?q=timekeeper|7,177|00:06|`#`|
|case 1|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:03|`#`|
|case 1|https://search.yahoo.co.jp/search?p=timekeeper|21,026|00:04|`#`|
|case 2|https://www.google.com/search?q=timekeeper|7,189|00:03|`#`|
|case 2|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:04|`#`|
|case 3|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:01|`#`|
|Average|-|5,976|00:03| |
The unit of size is bytes
The format of duration is "minutes:seconds"
one # represents 10 seconds in the duration graph
Input CSV file is here:
https://www.google.com/search?q=timekeeper,timekeeper_google.png
https://duckduckgo.com/?q=timekeeper&t=h_&ia=web,timekeeper_duckduckgo.png
https://search.yahoo.co.jp/search?p=timekeeper,timekeeper_yahoo.png
The test emits the following Markdown text:
## How long it took to navigate to URLs
as events flowed
|URL|duration|graph|
|:----|----:|:----|
|https://www.google.com/search?q=timekeeper|00:00|`#`|
|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|00:02|`#`|
|https://search.yahoo.co.jp/search?p=timekeeper|00:04|`#`|
|Average|00:02| |
The format of duration is "minutes:seconds"
one # represents 10 seconds in the duration graph
This Markdown text will be rendered on browser like this:
The code is here:
Javadoc is here
The artifact is available at the Maven Central repository:
Timekeeper was tested on Java8.
See build.gradle
at https://github.com/kazurayam/timekeeper/ for external dependencies.
As soon as you get a table generated by Timekeeper, you would feel like to sort it for better readability. Timekeeper is capable of it. Timekeeper supports sorting rows by multiple selected keys. It is possible to sort rows in either of ascending and descending order.
Let me show you a few examples of sorting rows.
Here I use a term “Attributes” to categorise the column names of a table such as “Case”, “URL”, etc. “Attributes” does not include the recorded numbers: size and duration.
You can find a sample code at
The sample code has this method:
@Test
void test_HTTPGetAndSaveResponse_sortByAttributes() {
Measurement interactions = new Measurement.Builder(
"get URL, save HTML into file", ["Case", "URL"])
.build()
// interact with URL, save the HTML into files
processURLs(urlList, outDir_, interactions)
// print the report
Timekeeper tk = new Timekeeper.Builder()
.table(new Table.Builder(interactions)
.sortByAttributes().build())
.build();
tk.report(outDir_.resolve("sortByAttributes.md"),
ReportOptions.NOLEGEND);
}
Please note the following fragment:
.sortByAttributes()
This fragment specifies Timekeeper to sort the rows by all Attributes of the table. If you have 2 or more attributes in the table, the left column has higher sorting priority than its right. In this example, the sorting key is : “Case” > “URL”.
Rows are sorted in ascending order unless the order is explicitly specified.
The output looks like this:
## get URL, save HTML into file
sorted by attributes (ascending)
|Case|URL|size|duration|graph|
|:----|:----|----:|----:|:----|
|case 1|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:04|`#`|
|case 1|https://search.yahoo.co.jp/search?p=timekeeper|21,026|00:05|`#`|
|case 1|https://www.google.com/search?q=timekeeper|7,189|00:04|`#`|
|case 2|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:03|`#`|
|case 2|https://www.google.com/search?q=timekeeper|7,153|00:06|`#`|
|case 3|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:03|`#`|
|Average|-|5,972|00:04| |
You can sort in descending order.
@Test
void test_HTTPGetAndSaveResponse_sortByAttributes_descending() {
Measurement interactions = new Measurement.Builder(
"get URL, save HTML into file", ["Case", "URL"])
.build()
// interact with URL, save the HTML into files
processURLs(urlList, outDir_, interactions)
// print the report
Timekeeper tk = new Timekeeper.Builder()
.table(new Table.Builder(interactions)
.sortByAttributes( RowOrder.DESCENDING ).build())
.build();
tk.report(outDir_.resolve("sortByAttributes_descending.md"),
ReportOptions.NOLEGEND);
}
Please note this fragment where you specify the descending order.
.sortByAttributes(Measurement.ROW_ORDER.DESCENDING).
The output looks like this:
## get URL, save HTML into file
sorted by attributes (descending)
|Case|URL|size|duration|graph|
|:----|:----|----:|----:|:----|
|case 3|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:01|`#`|
|case 2|https://www.google.com/search?q=timekeeper|7,177|00:02|`#`|
|case 2|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:03|`#`|
|case 1|https://www.google.com/search?q=timekeeper|7,165|00:05|`#`|
|case 1|https://search.yahoo.co.jp/search?p=timekeeper|21,026|00:03|`#`|
|case 1|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:03|`#`|
|Average|-|5,972|00:03| |
You can choose columns as sort key out of the Attributes.
@Test
void test_HTTPGetAndSaveResponse_sortByAttributes_URL() {
Measurement interactions = new Measurement.Builder(
"get URL, save HTML into file", ["Case", "URL"])
.build()
// interact with URL, save the HTML into files
processURLs(urlList, outDir_, interactions)
// print the report
Timekeeper tk = new Timekeeper.Builder()
.table(new Table.Builder(interactions)
.sortByAttributes(["URL"]).build())
.build()
tk.report(outDir_.resolve("sortByAttributes_URL.md"),
ReportOptions.NOLEGEND);
}
Please note the following fragment:
Measurement interactions = new Measurement.Builder(
"get URL, save HTML into file",
["Case", "URL"])
.sortByAttributes(["URL"]).
build();
The table has 2 attributes “Case” and “URL”. And you selected “URL” as the single sort key.
The output will look like this:
## get URL, save HTML into file
sorted by attributes (ascending)
|Case|URL|size|duration|graph|
|:----|:----|----:|----:|:----|
|case 1|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:02|`#`|
|case 2|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:05|`#`|
|case 3|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:02|`#`|
|case 1|https://search.yahoo.co.jp/search?p=timekeeper|21,026|00:05|`#`|
|case 1|https://www.google.com/search?q=timekeeper|7,189|00:06|`#`|
|case 2|https://www.google.com/search?q=timekeeper|7,197|00:06|`#`|
|Average|-|5,980|00:04| |
You can sort rows by duration.
@Test
void test_HTTPGetAndSaveResponse_sortByDuration_descending() {
Measurement interactions = new Measurement.Builder(
"get URL, save HTML into file", ["Case", "URL"]).build()
// interact with URL, save the HTML into files
processURLs(urlList, outDir_, interactions)
// print the report
Timekeeper tk = new Timekeeper.Builder()
.table(new Table.Builder(interactions)
.sortByDuration( RowOrder.DESCENDING ).build())
.build();
tk.report(outDir_.resolve("sortByDuration_descending.md"),
ReportOptions.NOLEGEND)
}
The output is like this:
## get URL, save HTML into file
sorted by duration (descending)
|Case|URL|size|duration|graph|
|:----|:----|----:|----:|:----|
|case 2|https://www.google.com/search?q=timekeeper|7,177|00:05|`#`|
|case 3|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:04|`#`|
|case 1|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:03|`#`|
|case 2|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:03|`#`|
|case 1|https://search.yahoo.co.jp/search?p=timekeeper|21,026|00:03|`#`|
|case 1|https://www.google.com/search?q=timekeeper|7,177|00:02|`#`|
|Average|-|5,974|00:03| |
You can sort rows by Attributes first, then secondly by duration. Perhaps this is the most useful way of sorting a Timekeeper’s table.
@Test
void test_HTTPGetAndSaveResponse_sortByAttributesThenDuration() {
Measurement interactions = new Measurement.Builder(
"get URL, save HTML into file", ["Case", "URL"]).build()
// interact with URL, save the HTML into files
processURLs(urlList, outDir_, interactions)
// print the report
Timekeeper tk = new Timekeeper.Builder()
.table(new Table.Builder(interactions)
.sortByAttributes().thenByDuration().build())
.build();
tk.report(outDir_.resolve("sortByAttributesThenDuration.md"),
ReportOptions.NOLEGEND)
}
The output looks like this:
## get URL, save HTML into file
sorted by attributes then duration (ascending)
|Case|URL|size|duration|graph|
|:----|:----|----:|----:|:----|
|case 1|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:02|`#`|
|case 1|https://search.yahoo.co.jp/search?p=timekeeper|21,026|00:02|`#`|
|case 1|https://www.google.com/search?q=timekeeper|7,165|00:05|`#`|
|case 2|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:02|`#`|
|case 2|https://www.google.com/search?q=timekeeper|7,177|00:05|`#`|
|case 3|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:01|`#`|
|Average|-|5,972|00:03| |
You can sort rows by size, of course.
@Test
void test_HTTPGetAndSaveResponse_sortBySize_ascending() {
Measurement interactions = new Measurement.Builder(
"get URL, save HTML into file", ["Case", "URL"]).build()
// interact with URL, save the HTML into files
processURLs(urlList, outDir_, interactions)
// print the report
Timekeeper tk = new Timekeeper.Builder()
.table(new Table.Builder(interactions)
.sortBySize().build())
.build();
tk.report(outDir_.resolve("sortBySize_ascending.md"),
ReportOptions.NOLEGEND);
}
The output is like this:
## get URL, save HTML into file
sorted by size
|Case|URL|size|duration|graph|
|:----|:----|----:|----:|:----|
|case 1|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:04|`#`|
|case 2|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:01|`#`|
|case 3|https://duckduckgo.com/?q=timekeeper&t=h_&ia=web|156|00:01|`#`|
|case 1|https://www.google.com/search?q=timekeeper|7,189|00:03|`#`|
|case 2|https://www.google.com/search?q=timekeeper|7,189|00:03|`#`|
|case 1|https://search.yahoo.co.jp/search?p=timekeeper|21,026|00:03|`#`|
|Average|-|5,978|00:03| |
RecordComparator
The Table.Builder
class implements 3 `sortBy*()`methods:
.sortByAttributes(List<String>, RowOrder)
.sortByDuration(RowOrder)
.sortBySize(RowOrder)
These can be followed by one or more thenBy*()
methods:
.thenByAttributes(List<String>, RowOrder)
.thenByDuratio(RowOrder)
.thenBySize(RowOrder)
.thenByAttributes(List<String>, RowOrder)
.thenByDuratio(RowOrder)
.thenBySize(RowOrder)
You can make a chain of multiple RecordComparators
. It is possible to chain 3 or more RecordComparators while specifying RowOrder.ASCENDING
and RowOrder.DESCENDING
to each comparators. For example, in Groovy, you can write:
Measurement m = new Measurement.Builder("ID",
["Case", "URL"]).build()
...
Table t = new Table.Builder(m)
.sortByAttribute(["URL"], RowOrder.ASCENDING)
.thenByAttribute(["Case], RowOrder.ASCENDING)
.thenByDuration(RowOrder.DESCENDING)
.build()
Please have a look at the source code of
The default format of Timekeeper report contains a few portions that may look verbose to you. You can opt them off. The options include:
the legend of table
the description how rows are sorted
the duration graph
@Test
void demo_noLegend() {
Measurement m1 =
new Measurement.Builder("How long it waited", ["Case"])
.build()
doRecording(m1)
Timekeeper tk = new Timekeeper.Builder()
.table(new Table.Builder(m1).build())
.build()
tk.report(outDir_.resolve("noLegend.md"),
ReportOptions.NOLEGEND);
}
Please note the line of .noLegend()
output:
## How long it waited
as events flowed
|Case|duration|graph|
|:----|----:|:----|
|sleeping for 13 secs|00:13|`##`|
|sleeping for 3 secs|00:03|`#`|
|sleeping for 7 secs|00:07|`#`|
|Average|00:07| |
Please note that there is no legend printed here.
@Test
void demo_noDescription() {
Measurement m1 =
new Measurement.Builder("How long it waited", ["Case"])
.build()
doRecording(m1)
Timekeeper tk = new Timekeeper.Builder()
.table(new Table.Builder(m1).build())
.build()
tk.report(outDir_.resolve("noDescription.md"),
ReportOptions.NODESCRIPTION)
}
Please note the line of .noDescription()
## How long it waited
|Case|duration|graph|
|:----|----:|:----|
|sleeping for 13 secs|00:13|`##`|
|sleeping for 3 secs|00:03|`#`|
|sleeping for 7 secs|00:07|`#`|
|Average|00:07| |
The format of duration is "minutes:seconds"
one # represents 10 seconds in the duration graph
Please note that there is no description like “sorted by duration (ascending)” printed.
@Test
void demo_noGraph() {
Measurement m1 =
new Measurement.Builder("How long it waited", ["Case"])
.build()
doRecording(m1)
Timekeeper tk = new Timekeeper.Builder()
.table(new Table.Builder(m1).build())
.build()
tk.report(outDir_.resolve("noGraph.md"),
ReportOptions.NOGRAPH);
}
Please note the line of .noGraph()
here.
## How long it waited
as events flowed
|Case|duration|
|:----|----:|
|sleeping for 13 secs|00:13|
|sleeping for 3 secs|00:03|
|sleeping for 7 secs|00:07|
|Average|00:07|
The format of duration is "minutes:seconds"
one # represents 10 seconds in the duration graph
Here there is no column of “graph”.
You can call .noDescription()
, .noLegend()
and .noGraph()
together.
@Test
void demo_the_simplest() {
Measurement m1 =
new Measurement.Builder("How long it waited", ["Case"])
.build()
doRecording(m1)
Timekeeper tk = new Timekeeper.Builder()
.table(new Table.Builder(m1).build())
.build()
tk.report(outDir_.resolve("the_simplest.md"),
ReportOptions.NODESCRIPTION_NOLEGEND_NOGRAPH)
}
Then you will get output as follows, which has the simplest format that Timekeeper can print.
## How long it waited
|Case|duration|
|:----|----:|
|sleeping for 13 secs|00:13|
|sleeping for 3 secs|00:03|
|sleeping for 7 secs|00:07|
|Average|00:07|
This is the simplest format that Timekeeper can print.
Timekeeper generates a report in Markdown text format as default. Optionally you can generate a report in CSV format.
You want to call another method Timekeeper#reportCSV(Path)
import com.kazurayam.timekeeper.Timekeeper
...
@Test
void demo_with_selenium_report_CSV() {
Timekeeper tk = runSeleniumTest();
tk.reportCSV(outDir_.resolve("report.csv"))
}