Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4267 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.3690. Min Act: -0.0000

Data Index: 1881932 (The Pile)

Max Activating Token Index: 541

Text #1

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.4042. Min Act: -0.0000

Data Index: 1551770 (The Pile)

Max Activating Token Index: 693

Text #2

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.4311. Min Act: -0.0000

Data Index: 720176 (The Pile)

Max Activating Token Index: 527

Text #3

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.4326. Min Act: -0.0000

Data Index: 823641 (The Pile)

Max Activating Token Index: 231

Text #4

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.4159. Min Act: -0.0000

Data Index: 1821923 (The Pile)

Max Activating Token Index: 89

Text #5

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.3624. Min Act: -0.0000

Data Index: 751895 (The Pile)

Max Activating Token Index: 149

Text #6

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.3543. Min Act: -0.0000

Data Index: 1883951 (The Pile)

Max Activating Token Index: 461

Text #7

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.4024. Min Act: -0.0000

Data Index: 1361543 (The Pile)

Max Activating Token Index: 1010

Text #8

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.4003. Min Act: -0.0000

Data Index: 883905 (The Pile)

Max Activating Token Index: 624

Text #9

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.3835. Min Act: -0.0000

Data Index: 48710 (The Pile)

Max Activating Token Index: 1018

Text #10

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.3690. Min Act: -0.0000

Data Index: 1966249 (The Pile)

Max Activating Token Index: 242

Text #11

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.3281. Min Act: -0.0000

Data Index: 106948 (The Pile)

Max Activating Token Index: 538

Text #12

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.3370. Min Act: -0.0000

Data Index: 1621409 (The Pile)

Max Activating Token Index: 716

Text #13

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.3970. Min Act: -0.0000

Data Index: 430276 (The Pile)

Max Activating Token Index: 204

Text #14

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.3544. Min Act: -0.0000

Data Index: 1165626 (The Pile)

Max Activating Token Index: 23

Text #15

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.4275. Min Act: -0.0000

Data Index: 184911 (The Pile)

Max Activating Token Index: 485

Text #16

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.3466. Min Act: -0.0000

Data Index: 1620192 (The Pile)

Max Activating Token Index: 176

Text #17

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.2619. Min Act: -0.0000

Data Index: 1792978 (The Pile)

Max Activating Token Index: 909

Text #18

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.3445. Min Act: -0.0000

Data Index: 1782085 (The Pile)

Max Activating Token Index: 160

Text #19

Max Range: 0.4326. Min Range: -0.4326

Max Act: 0.3086. Min Act: -0.0000

Data Index: 472007 (The Pile)

Max Activating Token Index: 113