HOW TO MAKE POKERTRACKER FLY . . . . (well almost)

General discussion of PokerTracker 3.

Moderator: Moderators

HOW TO MAKE POKERTRACKER FLY . . . . (well almost)

Postby Germaniac2 » Tue Aug 24, 2010 6:53 pm

-----
Over the past couple of weeks I have upgraded my system and experimented with various settings in the postgresql.conf file. The first two posts in this thread detail the hardware used, procedures employed, values tested, the results obtained, and finally the observations I have made along the way. A lot of this first post is a bit bloated, but it was easier to simply transfer all of my notes.

:idea: You may just want to skip most of the bottom of this post and concentrate on the salient parts in the second post. :idea:

Beyond these posts, I'd like to read your input and results that you have obtained in your own testing. I think that just a little more active involvement of the community of PokerTracker 3 users can lead to making this software perfom better and get ever closer to its full potential.

Please understand that while the final settings I used for my system worked well for me; these settings for the postgresql.conf file may not work for you at all. Rather than looking at these posts as a specific how to, think of them as a guideline for one possible approach to finetune your own setup and to maybe give you the impetus to enact some hardware changes and testing of your own.

I also felt that sharing my results may give some of you an insight into what is possible (performance wise) with PokerTracker 3 right now. While my completed system is nothing you can buy at your local electronics store or simply order online, all of the hardware used is of the shelf and easily available.

Other than my SSD drives all of my hardware is almost a year old, including the MegaRAID controller that I bought used on eBay at 1/4 of the new price. None of the components are exotic or high end, with the possible exception of the SSD drives. My CPU for example currently costs about $180. It is true that the combined costs of the components in my system exceed the cost of two high end store-bought systems. However, almost half of that is in the four SSDs alone. But when you gauge the performance achieved, and consider the value of the time saved by using such a setup, you may find the added expense quite reasonable and a worthwile investment.

As for building your own computer; it's fairly simple and at least to me is always a very satisfying experience (typically after some initial frustration and cursing). If you have never built your own computer, just do some research first, or ask a friend for some advice and/ or help.

The two initial posts are:

I.) Testing Notes, Preliminary Testing Log (This Post)
II.) Final Testing, Results, and Observations


--------------------------------------------------------------------------------------------------------
:arrow:-- T E S T I N G -- N O T E S :
--------------------------------------------------------------------------------------------------------

All Tests performed on HAL-MMX:------------------------------------08/13-20/2010 by MV

FoxConn Destroyer MB, AMD Phenom II x4 Deneb Black Edition 965 CPU (Quad-Core, AM3),
8GB OCZ Reaper RAM, CPU & RAM @ Defaults [3.4GHz, 1.0V | DDR2-AUTO, 1.8V, 5-5-5-12]
[Motherboard has nVIDIA onboard graphics + 4 PCIE x16 slots: The LSI MegaRAID card
is in slot #4, other slots used by 3 XFX GTS 250 1GB nVIDIA based graphics cards.
7 monitors (20"-28") are used (a 22" switchable to my notebook), plus 1 HDTV (#8)]

Boot Drive (C:\): 4 OCZ Vertex 2 SSDs, Controller: LSI MegaRAID 8888ELP 512MB Cache,
in RAID 0 Array (Direct I/O - No Disk Cache - Write Through - 577MB/s Tested Speed)

Secondary Drive (K:\): 2 WD VelociRaptor 10k RpM HDDs in RAID 0 Array, controlled by
nVIDIA 780a SLI (Chipset on MB). 191MB/s Tested Speed (Crystal DiskMark 3.64)
Copying (K:\) > (C:\) works at 11.2GB/Min. A 128GB Database transfers in 11.5 Mins

OS: Windows 7 (64-bit) Ultimate; PostgreSQL Version: 8.4.4; PT3 Version: 3.07.2 Beta
Files used for imports with equal structures and similar sizes (30kH-75kH) per run.
[Tests performed before Windows 7 service structure cleanup & tossing out the trash.]

PokerTracker3 Startup Times:-----11.00 Seconds from Icon click to running mode, i.e.
Holdem/Cash/General Tabs open, with 25.8k Players displayed in 354k Hand database.
20.50 Seconds from Icon click to running mode, i.e. Holdem/Cash/General Tabs Open,
with 82.4k Players displayed in 7.06M Hand database.

--------------------------------------------------------------------------------------------------------

PSQL Wiki Article at: http://wiki.postgresql.org/wiki/Tuning_ ... SQL_Server

Tests/ Changes performed here based on this Article & Modifications by Tuning Wizard.
FILE MODIFIED BY TUNING WIZARD (Ded. Server) DID NOT WORK ! (Server would not start.)
Probable Cause: PostgreSQL 8.4.4. was unable to understand the Memory Units inserted
by Tuning Wizard. "The Service did not report an error . . see . . NET HELPMSG 3534."

Tuning Wizard File CORRECTED and re-tested afterwards (see >>> Tests for results).

--------------------------------------------------------------------------------------------------------
:arrow:--T E S T I N G -- L O G :-----------------------------[Server Restarted for each Test]
--------------------------------------------------------------------------------------------------------
PRELIMINARY TESTS

08/13/2010 First Baseline Tests with Customized File (postgresql.conf):
shared_buffers = 512MB-----work_mem = 256MB-----maintenance_work_mem = 512MB
wal_buffers, checkpoint_segments, autovacuum, effective_cache - all at default values
Configuration works at ~420H/s (K:\) | ~408H/s (C:\) - NOTE: MAIN DRIVE IS SLOWER !

shared_buffers = 1024MB-----work_mem = 256MB-----maintenance_work_mem = 512MB
wal_buffers, checkpoint_segments, autovacuum, effective_cache - all at default values
Configuration runs @ ~140H/s (K:\) Imp. Speed, with VACUUM ERROR after Housekeeping.
[NEWER FILES (07/08/2010+): Structure must be different somehow! - INVESTIGATE]

shared_buffers = 512MB-----work_mem = 256MB-----maintenance_work_mem = 512MB
wal_buffers, checkpoint_segments, autovacuum, effective_cache - at default values
Configuration works at ~153H/s (K:\) | ~148H/s (C:\) Improved Speeds, NO ERRORS

Settings for shared_buffers above 512MB apparently cause problems. Also, PSQL Wiki
refers to 512MB as upper useful limit in Windows OS. [OTHER THAN THREE BLANK LINES
SEPARATING HANDS IN THE NEWER FILES COMPARED TO ONE LINE IN OLD FILES AND MORE
HANDS PER FILE (Same relation of size in MB to number of actual hands.), THERE IS NO
OTHER APPARENT DIFFERENCE BETWEEN OLDER AND NEWER FILES, THIS TEST SEEMS TO
CONFIRM THAT A PROBLEM WITH THE NEWER FILES MAY EXIST. FURTHER INVESTIGATION
WILL BE PERFORMED UPON COMPLETION OF ALL TESTS. MEANWHILE ONLY FILES MADE
BEFORE 07/08/2010 WILL BE USED. COULD THE TWO ADDITIONAL BLANK LINES PER HAND
CAUSE SUCH A DRAMATIC DIFFERENCE?]

As Above: w. wal_buffers = 1MB [4x Tuning Wizard (Ded.) Value | 16x Default Value]
Configuration runs @ ~475H/s (K:\) | ~470H/s (C:\) Imp. Speeds, NO ERRORS; This run
with an older file appears to prove the point made above, as it imported 275% faster
than the two previous tests. However, compared to Baseline Test, the wal_buffers may
be responsible for a 13% Gain. Further tests with wal_buffers to follow.

As Above: w. wal_buffers = 4MB [16x Tuning Wizard (Ded.) Value | 64x Default Value]
Configuration runs @ ~491H/s (K:\) Imp. Speed, wal_buffers responsible for 17% Gain.

As Above: w. wal_buffers = 16MB [64x Tuning Wizard (Ded.) Value | 256x Default Value]
AND temp_buffers at 32MB [4x Default Value], Maint. Run 13:10
Configuration runs @ ~483H/s (K:\) Import Speed, No Additional Gain.

As Above: w. wal_buffers = 16MB [64x Tuning Wizard (Ded.) Value | 256x Default Value]
AND temp_buffers at 32MB [4x Default Value], Maint. Run 13:15
Configuration runs @ ~457H/s (K:\) Import Speed, SLOWER/ reducing wal_buffers to 8MB.

As Above: w. wal_buffers = 8MB [32x Tuning Wizard (Ded.) Value | 128x Default Value]
PLUS: maintenance_work_mem = 1024MB [2x previous tests], Maint. Run ~13.5 Min.
Configuration runs @ ~476H/s (K:\) Imp. Speed, No Gain, Rolling back to prev. value.

As Above: w. effective_cache_size = 1024MB [~2x Tuning Wizard (Ded.) | 8x Default]
Configuration runs @ ~477H/s (K:\) Import Speed, No Gain, Maint. Run 13:00

As Above: w. effective_cache_size = 2048MB [~4x Tuning Wizard (Ded.) | 16x Default]
Configuration runs @ ~477H/s (K:\) Imp.Speed, No Gain, Maint. Run 12:50; Slight Gain

As Above: with wal_sync_method = open_sync [per PostgreSQL Wiki]
SERVER WILL NOT RESTART - REVERTED TO DEFAULT

As Above: w. checkpoint_segments = 64 [= Tuning Wizard (Ded.) | 21x Default (3)]
Configuration runs @ ~449H/s (K:\) Import Speed, 6% Slower, Maint. Run 13:05

As Above: w. checkpoint_segments = 32 [= 1/2 Tuning Wizard (Ded.) | 10x Default (3)]
AND checkpoint_completion_target = 0.9 [Per PostgreSQL Tuning Wiki; Default is 0.5]

Config. runs @ ~485H/s (K:\) Imp. Speed, 3% Faster, Maint. Run 16:40 - 27% SLOWER?
Effect of the increasing size of database? (Re-try prev. settings +Larger Eff. Cache)

As Above: w. wal_buffers = 12MB [48x Tuning Wizard (Ded.) Value | 192x Default Value]
AND checkpoint_segments = 24 [= 1/3 Tuning Wizard (Dedicated) | 8x Default (3)]
AND checkpoint_completion_target = 0.7 [Per PostgreSQL Tuning Wiki | Default is 0.5]
Configuration works at ~483H/s (K:\), Maint. Run 16:12 - Slightly Faster

PURGE RUN of 131 Sessions (G2): 8:50; Vacuum, Analyze & Rebuild Cache Run 45:55
Database (Observed) with 1.36M Hands after Purge.

As Above: with checkpoint_completion_target = 0.5 [Default]
Configuration runs @ ~456H/s (K:\) Import Speed, Maint. Run 18:15

Re-tried with checkpoint_segments = 64; checkpoint_completion_target = Default [0.5]
Configuration works at ~489H/s (K:\) Import Speed, Maint. Run 17:00 - Slightly Faster

Re-tried w. checkpoint_segments @ Default & checkpoint_completion_target @ Default
Configuration works at ~477H/s (K:\) Import Speed, Maint. Run 18:30 - Slower Again

As Above: w. checkpoint_segments = 128; checkpoint_completion_target = Default [0.5]
Configuration runs @ ~472H/s (K:\) Imp. Speed (files had 24 errors), Maint. Run 17:25

As Above: w. checkpoint_segments = 128; checkpt_completion_target = 0.7 [Def. = 0.5]
Configuration runs @ ~503H/s (K:\) Import Speed (files w. 9 errors), Maint. Run 17:35

As Above: w. checkpoint_segments = 128; checkpt_completion_target = 0.9 [Def. = 0.5]
Configuration runs @ ~497H/s (K:\) Imp. Speed (files w. 10 errors), Maint. Run 18:15

--------------------------------------------------------------------------------------------------------
PRELIM. TESTS W. CORRECTED TUNING WIZARD (Dedicated) FILE (All Tests Marked >>>)

>>> Corrections Made: Converted TUNING WIZARD's memory values to closest MB figures.
>>>
>>> shared_buffers = 152MB, work_mem = 42MB, maintenance_work_mem = 256MB
>>> effective_cache_size = 500MB, checkpoint_segments = 64, random_page_cost = 3.5
>>> autov_vacuum_threshold = 1000, autov_analyze_threshold = 250 [8.4.4 Defaults]
>1> Config. runs @ ~480H/s (K:\) Imp. Speed (files had 1 error), Maint. Run 15:45
>>> Import Speed Slightly Slower (4%) >>> BUT MAINT. RUN IS SHORTER BY 14% !
>2> Config. runs @ ~522H/s (K:\) Import Speed (files had 10 errors), Maint. Run 18:00
>>> Import Speed 10% FASTER AND MAINT. RUN IS SLIGHTLY SHORTER (2.7%)
>3> Config. runs @ ~525H/s (K:\) Import Speed (files had 7 errors), Maint. Run 19:10
>>> Import Speed Slightly Faster(1%) >>> BUT MAINT. RUN IS SLOWER BY 5%
>>> TRY random_page_cost modification first, then re-try different memory settings.

--------------------------------------------------------------------------------------------------------

BASED ON TESTING W. TUN-WIZ FILE: 152MB/42MB/256MB/500MB vs.512MB/256MB/512MB/2048MB
TESTS BELOW W. MEMORY VALUES HIGHER THAN TUN-WIZ's, BUT DIFFERENT FROM PREV. VALUES:

sh_buff = 456MB, wk_mem = 128MB, maint_work_mem = 768MB, eff._cache = 1536MB [TW x 3]
Config. runs @ ~526H/s (K:\) Import Speed (files had 5 errors), Maint. Run 14:30
VAST IMPROVEMENT IN MAINT. >> BUT RETURNS VACUUM ERROR ON COMPLETION OF HOUSEKEEPING.
(Improvement is probably the result of vacuum function being skipped.)

TEST RE-RUN W. REDUCTION IN maint_work_mem: 512MB APPEARS TO BE UPPER USEABLE LIMIT !
shared_buffers = 512MB, work_mem = 128MB, maint_work_mem = 512MB, eff._cache = 1536MB
Configuration runs @ ~510H/s (K:\) Imp. Speed (files had 37 errors), Maint. Run 18:30
2nd TEST RE-RUN WITH REDUCTION IN maintenance_work_mem sh._buffers = 512MB,
work_mem = 128MB, maint._work_mem = 512MB, eff._cache = 1536MB
Config. works at ~501H/s (K:\) Import Speed (files had 11 errors), Maint. Run 18:00

TEST BELOW WITH MEMORY VALUES ~2.5 x TUNING WIZARD's NUMBERS (except maint._work_mem)
sh._buffers = 384MB, work_mem = 96MB, maint._work_mem = 512MB, eff._cache = 1280MB
Config. runs at ~500H/s (K:\) Import Speed (files had 9 errors), Maint. Run 17:15

>>> Test >3> (TW Corrected) from above rerun for verification at this point. OK

---------------------------------------------------------------------------------------------------------
PURGED 56 Sessions (G2) w. Test >3> Settings: 8:48; Vac., An. & Reb. Cache - 1:00:53
Database (Observed): 1.83M Hands after Purge. CORRECTED TUNING WIZARD SPEEDS LOOK OK

--------------------------------------------------------------------------------------------------------
08/16/2010 RE-RUN TEST W. BEST RESULTS SO FAR TO CONFIRM VACUUM ERROR OCCURS AGAIN.
sh._buff = 512MB, wk_mem = 128MB, mt_work_mem = 768MB, eff._cache = 1536MB [TW x 3]
CONFIGURATION WILL NOT RESTART (No Error - As Before) - SETTINGS CHANGED TO:
shared_buffers = 512MB, work_mem = 96MB, maint_work_mem = 640MB, eff._cache = 1408MB
CONFIGURATION STILL DOES NOT RESTART (Still No Error) - SETTINGS CHANGED TO:
shared_buffers = 512MB, work_mem = 96MB, maint_work_mem = 512MB, eff._cache = 1280MB
CONFIGURATION STILL DOES NOT RESTART ! RUN FURTHER TESTS ON TW FILE, THEN FIND CAUSE.

--------------------------------------------------------------------------------------------------------
>4> 08/16/2010 Tuning Wizard Conf. File Retested to obtain current comparison values.
>>> Config. runs @ ~457H/s (K:\) Import Speed (files had 13 errors), Maint. Run 27:01
>>> Import Speed 10% Slower Than Yesterday. MAINTENANCE RUN TIMES STILL INCREASING.
>>> Machine is running 61 services, investigate which addl. services are running.
>>> (Probably Maintenance Tasks. Find & shut them down to obtain valid test numbers.)
>>>
>5> Tuning Wizard Conf. Tested w. 3x Memory Values: 384/128/768/1536 +1MB wal_buffers
>>> ATTEMPT TO CREATE THE SAME ERROR THAT OCCURS USING THIS FILE AT SAME SETTINGS.
>>> Config. works at ~482H/s (K:\) Import Speed (files w. 9 errors), Maint. Run 14:30
>>> VACUUM ERROR !!! Import Speed 5.5% BETTER THAN >4>. MAINT. RUN IS SHORTER BY 40%
>>>
>6> Config. >5> with Lower maint_work_mem Value: 384/128/512/1536 + 1MB wal_buffers
>>> Config. runs @ at ~498H/s (K:\) Imp. Speed (files had 7 errors), Maint. Run 19:30
>>> NO ERRORS !!! Import Speed 3% BETTER THAN >5>. MAINTENANCE RUN OK
>>> TESTS >5> & >6> CONFIRM THAT SETTING maint_work_mem ABOVE 512MB RESULTS IN ERROR
>>>
>7> Configuration >5> with increased wal_buffers: 384/128/512/1536 + 4MB wal_buffers
>>> Config. runs @ ~481H/s (K:\) Imp. Speed (files had 16 errors), Maint. Run 19:31
>>> NO ERRORS - Import Speed same as >5>. Maintenance Run = >6>
>B> Second Run: ~509H/s (K:\) Import Speed (files had 11 errors), Maint. Run 19:40
>>>
>8> Further increase to 8MB in wal_buffers: 512/128/512/1536 + 24MB temp_buffers
WOW Config. works at ~594H/s (K:\) Imp. Speed (files w. 14 errors), Maint. Run 20:30
>B> Second Run: ~561H/s (K:\) Import Speed (files had 9 errors), Maint. Run 20:30
>>> Apparently adding a decent amount of temp_buffers (wal x3) in addition to upping
>>> the amount of wal_buffers themselves works like a charm ! (~Avg. % Gain !)

##################################################################################
#--Based on PT3 Forum Post by kraada 06/19/2010, and email confirmation by Derek on-- #
#--08/17/2010 Cluster Function will be skipped from now on. (Not needed on SSDs.)------ #
##################################################################################

>9> Config. >8> increased wal_buffers to 12MB: 512/128/512/1536 + 32MB temp_buffers
>>> Config. works at ~547H/s (K:\) Imp. Speed (files had 5 errors), Maint. Run 7:35!
>B> Second Run: ~562H/s (K:\) Import Speed (files had x errors), Maint. Run 6:45!
>>> No Speed Gain vs. Test >8>, Database Now Contains 2.2 M Hands.
>>>
>10 wal_buffers red. by 2MB: 512/128/512/1536 + 10MB wal_buffers + 30MB temp_buffers
>>> Config. runs @ ~536H/s (K:\) Imp. Speed (files had 11 errors), Maint. Run 7:03
>B> Second Run: ~531H/s (K:\) Import Speed (files had 14 errors), Maint. Run 7:35
>>>
>>> PER POSTGRESQL WIKI: Upper useful limit of wal_buffers is 16MB = Next Tests
>>>
>11 512/128/512/2048 + 16MB wal_buffers + 32MB temp_buffers
>>> Config. runs @ at ~537H/s (K:\) Imp. Speed (files w. 15 errors), Maint. Run 6:40
>B> Second Run: ~531H/s (K:\) Import Speed (files had 13 errors), Maint. Run 6:20
>>> NO APPARENT SPEED GAIN (First Test Peaked at 614H/s) - Back to 8MB wal_buffers

--------------------------------------------------------------------------------------------------------
TEST SERIES BELOW: Performed w. HIGH LIMIT ($30/60+) hands, may make a difference in
that shorter hands/ no flop hands, take less space & should import at a faster rate.

>81 Config. RE-TESTED with 8MB wal_buffers: 512/128/512/2048 + 24MB temp_buffers
WOW Config runs @ ~675 H/s (K:\) Import Speed (files had 10 errors), Maint. Run 6:15
>B> Second Run: ~676H/s (K:\) Import Speed (files had 7 errors), Maint. Run 6:50
>C> Third Run: ~662H/s (K:\) Import Speed (files had 18 errors), Maint. Run 6:35
>>> PEAK IMPORT SPEED AT 948H/s ! ! !
>>>
>>> TESTS BELOW: Config. >81 with fsync_writethrough as synchronous commit setting
>>> Config. runs @ ~656H/s (K:\)Import Speed (files w. 11 errors), Maint. Run 7:30
>B> Second Run: ~629H/s (K:\) Import Speed (files had 5 errors), Maint. Run 6:35
>>> NO APPARENT SPEED GAIN - Rolling Back to previous synchronous commit setting.
>>>
>>> TESTS BELOW: Configuration >81 w. max_stack_depth increased to 8MB [Default x4]
>>> SERVER WILL NOT RESTART (No Error)
>>> ---> trying max_stack_depth increased to 4MB [Default x2]
>>> SERVER WILL NOT RESTART (No Error)
>>> ---> trying max_stack_depth reduced to only 1MB [1/2 Default]
>>> Config. runs @ ~597H/s (K:\) Import Speed (files had 2 errors), Maint. Run 4:20
>>>
>>> ---> trying max_stack_depth increased to 3MB [Default +50%]
>>> Config. works at ~650H/s (K:\) Imp. Speed (files had 8 errors), Maint. Run 6:25
>>> Second Run: ~610H/s (K:\) Import Speed (files had 27 errors), Maint. Run 7:00
>>>
>>> RESET max_stack_depth to 2MB [Default] for Comparison Test with same conditions.
>>> Config. works at ~655H/s (K:\) Imp. Speed (files had 13 errors), Maint. Run 5:40
>>> Second Run: ~646H/s (K:\) Import Speed (files had 2 errors), Maint. Run 4:00
>>> ---> LEAVE max_stack_depth at 2MB [Default]

--------------------------------------------------------------------------------------------------------
Now Using $6NL Files, Next 2 Tests w. above Config. (>81) to establish new baseline.
>>> Config. runs @ ~554H/s (K:\) Imp. Speed (files had 3 errors), Maint. Run 8:46
>>> Second Run: ~552H/s (K:\) Import Speed (files had 11 errors), Maint. Run 8:00
>>>
>>> ---> trying max_prepared_transactions increased to 10 [Default x2]
>>> Config. works at ~542H/s (K:\) Imp. Speed (files had 8 errors), Maint. Run 9:10
>B> Second Run: ~545H/s (K:\) Import Speed (files had 6 errors), Maint. Run 9:22
>>> Appears to have no significant Impact (2% slower), reverting to Default Setting.
>>>
>>> ---> TURNED OFF LOGGING
>>> Config. works at ~541H/s (K:\) Import Speed (files had 6 errors), Maint. Run 8:40
>B> Second Run: ~543H/s (K:\) Import Speed (files had 5 errors), Maint. Run 8:28
>>> NO GAIN FROM TURNING OFF LOGGING > > Logging Re-Enabled

PURGE of 71 Sessions (G2) [>8> Settings]: 39:15; Vac., Analyze & Reb Cache - 3:04:09
Database (Observed) has 3.45M Hands after Purge.

--------------------------------------------------------------------------------------------------------

COMPARISION OF TUNING WIZARD (Modified) & THIS FILE--------------------(08/18/2010)
To determine the cause(s) that will not allow a server restart with
THIS file. Differences in Formating / Missing or Added Symbols (#) ?
(Also: Increased effective_cache_size to 4096MB below.)

SEARCH RESULTS:
Some superfluous # symbols on far right, plus possibly a problem w.
previous formatting at 100 characters width, narrowed it to 90.

> > > FILE NOW WORKS AGAIN - Ready for Final Testing

--------------------------------------------------------------------------------------------------------
Germaniac2
 
Posts: 19
Joined: Fri Oct 16, 2009 2:30 pm

Re: HOW TO MAKE POKERTRACKER FLY . . . . (well almost)

Postby Germaniac2 » Tue Aug 24, 2010 8:08 pm

---PART II)
-------------------------------------------------------------------------------------------------------------
:arrow:-- F I N A L ---T E S T I N G:
-------------------------------------------------------------------------------------------------------------
Final Comparison Tests run with ORIGINAL postgresql.conf and CORRECTED Tuning
Wizard Files. RESULTS Figures are averages of 3 runs per Configuration with ~200,000
Hands per test. Each run series conducted sequentially, in alternating orders for better
accuracy; i.e: Run 1 ($1NL): A-B-C-D-E; Run 2 ($2NL): E-D-C-B-A; Run 3 ($4NL): C-B-E-A-D
Server was restarted for each Test, Machine Rebooted for D) and E) series Tests.

In addition to PokerTracker 3 the following programs were running for all Tests:
Task Manager, 1 Explorer Window, 1 Notepad Window. Microsoft Security Essentials
and Windows Firewall were running in the background. Total Services running: 57.

Before Run 1, Database had 3.45 Million Hands; each runs adds ~1 Million Hands.
Figures after (Test Letter): xxx Hands/sec. Import Speed | mm:ss Maint. Run Time
CPU/RAM figures show MAXIMUM resources used at any time during the testing.

--Run 1:--------195,334 H-------196,379 H-------197,013 H-------195,629 H------197,011 H
--($1NL)----A: 391|25:50---B: 476|26:10---C: 532|26:10---D: 546|22:45---E: 569|22:20
-CPU/RAM-----62%|1.3GB-----48%|1.9GB------52%|2.2GB------44%|2.2GB-----39%|2.2GB

--Run 2:--------202,037 H-------200,346 H-------201,695 H-------200,667 H------202,639 H
--($2NL)----A: 400|23:45---B: 450|21:10---C: 530|21:15---D: 564|21:10---E: 558|20:15
-CPU/RAM-----58%|1.2GB-----42%|1.9GB------47%|2.3GB------42%|2.0GB-----56%|2.2GB

--Run 3:--------201,637 H-------206,139 H-------206,069 H-------200,631 H------200,509 H
--($4NL)----A: 394|23:30---B: 458|19:55---C: 524|20:20---D: 541|22:45---E: 559|22:20
-CPU/RAM-----56%|1.4GB-----39%|1.7GB------42%|2.3GB------39%|2.2GB-----65%|2.4GB

Final Test Series started on 08/18/2010 at 15:05, completed on 08/19/2010 at 18:40
Break after 8 of 15 Tests, Database at 5.03 Million Hands.

After Final Test Runs:
PURGING of 5 Sessions (G2-Setting D):-------6m:38s-------Housekeeping Run: 5:07:45 **
Database (Observed) has 7.06M Hands after Purge.----(Vacuum, Analyze, Rebuild Cache)

** NOTE:
When manually importing hands, particularly datamined hands that include only a few
sessions you have played, you may be able to save a lot of time by first purging your
played hands, PRIOR to running housekeeping. This eliminates the need to rebuild the
cache which can take quite a long time. You'll still need to vacuum, analyze, (cluster
if your database is on HDD), and UPDATE the cache. But that's MUCH faster.
However, this doesn't apply to large manual imports in the millions of hands.

-------------------------------------------------------------------------------------------------------------
:arrow:-- R E S U L T S:
-------------------------------------------------------------------------------------------------------------

A) ORIGINAL PSQL.CONF FILE:-----~395H/s (K:\) Import Speed, Maintenance Run--24m:22s

B) TUNING WIZARD (Ded.)FILE:----~461H/s (K:\) Import Speed, Maintenance Run--22m:25s

C) MV MODIFIED FILE:----------------~529H/s (K:\) Import Speed, Maintenance Run--22m:32s

D) MV MOD.FILE OC GEAR 1:--------~550H/s (K:\) Import Speed, Maintenance Run--22m:13s

E) MV MOD.FILE OC GEAR 2:--------~562H/s (K:\) Import Speed, Maintenance Run--21m:50s

(Average Import File Size ~256MB/~200k Hands | Vacuum, Analyze & Update Cache Only)


MAXIMUM PERFORMANCE GAINS:-------------------~ +167H/s----------~ -2m:32s
(Original File vs. OC2 MV Mod. File)----------------42.3% Faster-----11.6% Quicker

MAXIMUM EXPECTED TIME SAVINGS PER MILLION HANDS IMPORTED:----~24m:32s (15.2%)
(42m:11s vs 29m:39s Import + 2H:01m:50s vs 1H:49m:50s Maint.)

At Average of 1.5 Million Hands Imported per Week:---~32 Hours Time Savings per Year
(Savings do not account for improved purging speeds, faster HUD response, etc.)

Test with System Stable OverClocked Values of:
OC Gear 1: CPU @ 3.6GHz, 1.05V, CPU Stepping 1 - RAM @ 1066 MHz, 2.19V
OC Gear 2: CPU @ 3.8GHz, 1.08V, CPU Stepping 3 - RAM @ 1066 MHz, 2.23V (+1 Step)

CPU Temps (Max.):-----Stock - 42.4 C | OC Gear 1 - 44.6 C | OC Gear 2 - 46.3 C
CPU Temps (Typ.):-----Stock - 40.6 C | OC Gear 1 - 41.3 C | OC Gear 2 - 41.8 C
CPU Temps (Idle):------Stock - 34.8 C | OC Gear 1 - 35.6 C | OC Gear 2 - 37.2 C

-------------------------------------------------------------------------------------------------------------
:arrow:-- O B S E R V A T I O N S:
-------------------------------------------------------------------------------------------------------------
Using Tuning Wizard (dedicated server option) to update the postgreSQL.conf file is
a good start in improving PokerTracker 3 performance. However, be aware that you
may have to correct the values put in by Tuning Wizard to get it to work at all. This
test series showed an average improvement in import speeds of 16.7% compared to
the original file & a modest 8% reduction in the time taken for maintenance tasks.

Increasing memory settings will only help to a certain extent. Beyond the settings
tested here, PostgreSQL and therefore PokerTracker may just cease to work. A good
example for this is the value used for maintenance_work_mem: Values over 512MB
returned a Vacuum Error Message upon completion of housekeeping in my tests above.
Similarly, increasing the value of shared_buffers beyond 512MB did not result in any
additional performance gains for me and did return some errors. (The PostgeSQL Wiki
also refers to 512MB for this parameter as the upper useful limit in Windows.)

This test series also shows the value of experimenting with your settings. I was not
previously aware that a change in the temp_buffers setting would have any impact. I
did not find anything about temp_buffers in the PT forums, and the PSQL Wiki is also
mum on that subject. Yet, when you look at the test result for test >8> above you'll
see that adjusting the temp_buffers together with wal_buffers gave me major gains.

Increasing the size of wal_buffers made a big difference here. Try some different
settings and test what increasing this number to 4MB, 8MB or even 16MB will do for
you. (Per PSQL Wiki: 16MB is the upper useful limit and my tests confirmed it, and
I'll stick with 12MB for now.) Also try some different settings for temp_buffers.
Most likeley various hardware configurations will act quite different to tweaks.

Using SSDs (YES, I know they're not cheap!) not only gets you faster imports, but
also allows you to skip the clustering step in housekeeping. Since clustering takes
about 40-45% of the total time spent on housekeeping, significant chunks of time
can be saved through the use of SSDs. Since this alone can get your machine ready
for use much faster, you should seriously consider this option. If you could save an
average of only an hour a week by using SSDs and your time is only worth $12 per
hour, you'll almost be saving enough in a year for two 120GB OCZ Vertex 2 drives.

Maintenance chores will simply keep taking up more & more time as your database
grows. It's a fact of life, better get used to it. Also the speed at which PT3 goes
about purging hands is simply painfully slow. While I realize that there are many
relations that need to be terminated, I still fail to understand why a purge takes
many multiples of the time it took to import the hands and create these relations
in the first place.

ANYONE WITH A BRIGHT IDEA TO IMPROVE PURGING, PLEASE STEP FORWARD!

Primarily because of the above described purging dilemma I will be starting a new
rotation system with new databases created every quarter, as described in the PT3
tutorial on managing multiple databases. Though, I like keeping hands for different
lengths of time. (e.g.: 12 weeks is great for $1NL, but for LIMIT hands, especially
$10/$20 and above, I prefer keeping these hands active for up to 4 times as long.

Another bright spot: Now that Beta 4 of PostgreSQL 9.0 (64-bit), 'likely the last
Beta release', has been out for 2 weeks, 64-bit Windows database management is
around the corner. While it is unlikely that PokerTracker will release a full 64-bit
version any time soon, testing with the 64-bit version of PSQL 9.0 is being done by
the PT3 development team now. What we know so far is that PT3 does work with 9.0.

Where better performance for PT3 is concerned, more testing is needed. Because PSQL
is the 'engine' of PT3, I for one suspect that appreciable performance gains could
be possible here. (Think of it like putting a V6 into a trusty old VW Beetle.)

Why am I so upbeat on 64-bits? As many of you know, a 32-bit Windows OS can address
(work with) a maximum of 4GB in RAM. Most programs, including PSQL 8.x, use far less.
A 64-bit OS can theoretically address more than 128GB of RAM. However, most desktop
computers are limited by their motherboards to between 12GB and 24GB. But that still
is akin to switching from a regular TV to HDTV. Even with only 8GB to 12GB RAM your
system will produce noticeable differences in performance.

Of course, this all depends on how the developers of PSQL 9.0 have structured their
software and how much memory it will be able to address under a 64-bit Windows OS.
I for one have high hopes that the move to PSQL 9.0 will result in perfomance gains
that are at least similar in scale to what I achieved with my new hardware and tweaks
in the postgresql.config file in these tests. I think that a potential boost in the range
of to 25-40% in the performance of PSQL and therefore the speed PokerTracker 3
operates at is entirely possible. While this requires that you upgrade your OS to
64-bits (if haven't already) and maybe increasing your RAM to at least 6GB, I think
that this rather modest investment could be well worth it. Time & Testing will tell.

A bit more about hardware: As these results clearly show; OverClocking your system
will speed up the performance of PokerTracker a little. BUT, like with OverClocking
in general: BE CAREFUL. You could easily ruin your CPU or your Memory. Even worse,
a FATAL ERROR in your database might set you back days or longer. That being said,
these fairly mild OC values I used on my air-cooled system gave me an increase of
another 6.2% in the import performance of PokerTracker. Interestingly enough, while
none of the tweaks in the various versions of the postgresql.conf files tested here
seems to have sped up the housekeeping functions much, OverClocking appears to also
only help a small amount. On average, OverClocking shortened the time housekeeping
cycles took by 1.6% (OC-Gear 1) and 3.3% (OC-Gear 2) respectively.

Imports from my primary SSD drive system, which houses the OS & databases, worked
a little slower than importing hands from my backup/ file drives (WD Raptors/ RAID 0),
even though my primary array is more than 3x faster than the Raptors. Therefore I'd
suggest you make sure that you have a speedy backup drive (a single Raptor or SSD)
to download new hand files and import them from, while keeping your primary drive(s)
running the OS, PostgreSQL and PokerTracker. Also, making sure that you frequently
defragment that drive (if its HDD(s) not SSD(s)), as this also helps import speeds.
I run a defrag cycle after downloading new hands and before importing them and it
does make a noticable difference. (But only on import speed.)

LAST, BUT DEFINETLY NOT LEAST, file structures can have a far greater impact on
the time it takes to import them than anything else. My dataprovider recently had a
change, and as a result the files he provides me now are somehow different. While I
so far can see no apparent structural, or for that matter ANY other difference that
SHOULD matter, the newer files import at a speed that is less than 1/3 of what its
been taking for the older files to be imported. I WILL INVESTIGATE THIS
FURTHER & FIND THE CAUSE. Should the need arise I will have to switch providers.

So do yourself a favour, and get some files from various Hand History providers to
see whether some work better for you than others. Just make sure to use files for
the same Poker Site, Game Type, Game Limit, etc.; if you want valid test results.
Feel free to copy and paste the settings below for tests with your machine. Be aware
that just because they worked for me, doesn't mean that they will work for you. Also,
be careful when cutting and pasting: A missing or added character could prevent the
file from loading and / or running properly. (Been There - Done That) When testing
make sure you document your changes. After a few tests (or several dozen), its easy
to get confused, forget what combinations you have tried, or get into a real bind.
If you can re-trace your steps its pretty easy though. Have some fun with this!

Like it's been said in these forums and elsewhere on several occasions before:

"Tuning PostgreSQL for Performance is (somewhat) more of an Art than a Science."

-------------------------------------------------------------------------------------------------------------

Germaniac2
 
Posts: 19
Joined: Fri Oct 16, 2009 2:30 pm

Re: HOW TO MAKE POKERTRACKER FLY . . . . (well almost)

Postby bloodndef » Wed Aug 25, 2010 1:48 pm

Good post, thank you!

##################################################################################
#--Based on PT3 Forum Post by kraada 06/19/2010, and email confirmation by Derek on-- #
#--08/17/2010 Cluster Function will be skipped from now on. (Not needed on SSDs.)------ #
##################################################################################


Am I reading this right? Is this true?
bloodndef
 
Posts: 99
Joined: Fri Feb 27, 2009 1:28 am

Re: HOW TO MAKE POKERTRACKER FLY . . . . (well almost)

Postby zubs1aa » Wed Aug 25, 2010 1:56 pm

bloodndef wrote:Good post, thank you!

##################################################################################
#--Based on PT3 Forum Post by kraada 06/19/2010, and email confirmation by Derek on-- #
#--08/17/2010 Cluster Function will be skipped from now on. (Not needed on SSDs.)------ #
##################################################################################


Am I reading this right? Is this true?


I think he means that if you are using SSD there is no need to cluster. You will have to cluster though if you are using a hard drive however. That is how I took earlier posts in another thread on that subject.
zubs1aa
 
Posts: 2219
Joined: Fri Feb 08, 2008 1:52 pm

Re: HOW TO MAKE POKERTRACKER FLY . . . . (well almost)

Postby kraada » Wed Aug 25, 2010 2:58 pm

That is correct.

Also, to the OP: thanks for all of this, it's a great analysis!
kraada
Moderator
 
Posts: 54431
Joined: Wed Mar 05, 2008 2:32 am
Location: NY

Re: HOW TO MAKE POKERTRACKER FLY . . . . (well almost)

Postby zubs1aa » Wed Aug 25, 2010 3:14 pm

kraada wrote:That is correct.

Also, to the OP: thanks for all of this, it's a great analysis!


it is, but could we get some better cliffs from that thing? I like to think I'm technical enough, but I got lost. I'm sure there is a way to break it out to more relavant bullets somehow?
zubs1aa
 
Posts: 2219
Joined: Fri Feb 08, 2008 1:52 pm

Re: HOW TO MAKE POKERTRACKER FLY . . . . (well almost)

Postby kraada » Wed Aug 25, 2010 3:56 pm

I'll see what I can do.

Though the simplest cliffs are: SSDs rock and 8G of RAM is good :)
kraada
Moderator
 
Posts: 54431
Joined: Wed Mar 05, 2008 2:32 am
Location: NY

Re: HOW TO MAKE POKERTRACKER FLY . . . . (well almost)

Postby zubs1aa » Wed Aug 25, 2010 4:28 pm

kraada wrote:I'll see what I can do.

Though the simplest cliffs are: SSDs rock and 8G of RAM is good :)


DUH lol I have the SSD and as much ram as I can lol. I meant in terms of what the 4 or so settings are and how to change them.
zubs1aa
 
Posts: 2219
Joined: Fri Feb 08, 2008 1:52 pm

Re: HOW TO MAKE POKERTRACKER FLY . . . . (well almost)

Postby Germaniac2 » Wed Aug 25, 2010 4:53 pm

-
One way to simplify this a bit is to just start with the original
Performance Tuning PostgreSQL Tutorial by APerfect10 from October 2008:


http://www.pokertracker.com/forums/viewtopic.php?f=45&t=10393

In addition to this, I would recommend the following:
-
---• Run your OS, PT3, and your Database(s) from a fast SSD (or an SSD Array)
------Your system & PT3 will be faster. You will eliminate the need for clustering & save time.

---• Install a fast secondary drive (at least a 10k rpm HDD, or a fast SSD)
------This will help getting your imports done faster than running them from your OS drive.
------You will also have a safer place for your database backups if your primary drive(s) fail(s).

---• Increase the size of your wal_buffers and temp_buffers
------Your import speeds should improve considerably with this change.
------This is System Specific - You will need to test what works for you.

Most importantly keep records of all changes you make, so you can easily revert to a
last known good configuration. Beyond the few steps above there are lots of possibilities.
Settings for random_page_cost, checkpoint_segments, effective_cache_size, and many
others, either individually or in various combinations can all have an impact on the overall
performance of your system when runnung PokerTracker 3.

There are no hard & fast rules here. I am just a user of PokerTracker 3 myself; I have
no affiliation with either the PT3 development team or their (excellent) support.
However, the last thing I want to result from my posts is several guys screwing up their
systems and then bogging down support with entirely avoidable support requests. So if
you are not at ease in working with systems configuration files, you may just want to
pass on this one [Know when to fold!].

-
Germaniac2
 
Posts: 19
Joined: Fri Oct 16, 2009 2:30 pm

Re: HOW TO MAKE POKERTRACKER FLY . . . . (well almost)

Postby zubs1aa » Wed Aug 25, 2010 5:06 pm

Germaniac2 wrote:-
One way to simplify this a bit is to just start with the original
Performance Tuning PostgreSQL Tutorial by APerfect10 from October 2008:


http://www.pokertracker.com/forums/viewtopic.php?f=45&t=10393


---• Increase the size of your wal_buffers and temp_buffers
------Your import speeds should improve considerably with this change.
------This is System Specific - You will need to test what works for you.

Most importantly keep records of all changes you make, so you can easily revert to a
last known good configuration. Beyond the few steps above there are lots of possibilities.
Settings for random_page_cost, checkpoint_segments, effective_cache_size, and many
others, either individually or in various combinations can all have an impact on the overall
performance of your system when runnung PokerTracker 3.

There are no hard & fast rules here. I am just a user of PokerTracker 3 myself; I have
no affiliation with either the PT3 development team or their (excellent) support.
However, the last thing I want to result from my posts is several guys screwing up their
systems and then bogging down support with entirely avoidable support requests. So if
you are not at ease in working with systems configuration files, you may just want to
pass on this one [Know when to fold!].

-



I'm actually very comfortable working with files and have doen th etuning in the past... but it seems like I should re-examine. The part I didn't understand was what WalBuffers and temp_buffers / the other variales etc were in order to make decisions about how to play with them.

Essentially lookin for easy to read descriptions of the various settings that get played with and what they affect. (no offense, don't need the colors, just simple easy to read descriptions...)
zubs1aa
 
Posts: 2219
Joined: Fri Feb 08, 2008 1:52 pm

Next

Return to General [Read Only]

Who is online

Users browsing this forum: No registered users and 27 guests

cron