Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2068 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.1512. Min Range: -3.1512

Max Act: 3.1512. Min Act: -0.0000

Data Index: 186069 (The Pile)

Max Activating Token Index: 457

Text #1

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.9997. Min Act: -0.0000

Data Index: 1124973 (The Pile)

Max Activating Token Index: 315

Text #2

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.8161. Min Act: -0.0000

Data Index: 603145 (The Pile)

Max Activating Token Index: 610

Text #3

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.4316. Min Act: -0.0000

Data Index: 424802 (The Pile)

Max Activating Token Index: 452

Text #4

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.2452. Min Act: -0.0000

Data Index: 937529 (The Pile)

Max Activating Token Index: 392

Text #5

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.3729. Min Act: -0.0000

Data Index: 876371 (The Pile)

Max Activating Token Index: 709

Text #6

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.3670. Min Act: -0.0000

Data Index: 600862 (The Pile)

Max Activating Token Index: 282

Text #7

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.1507. Min Act: -0.0000

Data Index: 108433 (The Pile)

Max Activating Token Index: 920

Text #8

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.2952. Min Act: -0.0000

Data Index: 867213 (The Pile)

Max Activating Token Index: 208

Text #9

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.7187. Min Act: -0.0000

Data Index: 1008852 (The Pile)

Max Activating Token Index: 879

Text #10

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.2865. Min Act: -0.0000

Data Index: 252808 (The Pile)

Max Activating Token Index: 470

Text #11

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.1536. Min Act: -0.0000

Data Index: 584258 (The Pile)

Max Activating Token Index: 196

Text #12

Max Range: 3.1512. Min Range: -3.1512

Max Act: 1.9485. Min Act: -0.0000

Data Index: 31042 (The Pile)

Max Activating Token Index: 503

Text #13

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.2549. Min Act: -0.0000

Data Index: 1939134 (The Pile)

Max Activating Token Index: 937

Text #14

Max Range: 3.1512. Min Range: -3.1512

Max Act: 1.9925. Min Act: -0.0000

Data Index: 591116 (The Pile)

Max Activating Token Index: 768

Text #15

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.4277. Min Act: -0.0000

Data Index: 1250554 (The Pile)

Max Activating Token Index: 914

Text #16

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.1819. Min Act: -0.0000

Data Index: 1357233 (The Pile)

Max Activating Token Index: 25

Text #17

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.4155. Min Act: -0.0000

Data Index: 376144 (The Pile)

Max Activating Token Index: 625

Text #18

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.1599. Min Act: -0.0000

Data Index: 1463499 (The Pile)

Max Activating Token Index: 937

Text #19

Max Range: 3.1512. Min Range: -3.1512

Max Act: 2.0526. Min Act: -0.0000

Data Index: 1128796 (The Pile)

Max Activating Token Index: 329