Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2067 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 7.1346. Min Range: -7.1346

Max Act: 7.1346. Min Act: -0.0000

Data Index: 866108 (The Pile)

Max Activating Token Index: 363

Text #1

Max Range: 7.1346. Min Range: -7.1346

Max Act: 6.6596. Min Act: -0.0000

Data Index: 859533 (The Pile)

Max Activating Token Index: 315

Text #2

Max Range: 7.1346. Min Range: -7.1346

Max Act: 6.6434. Min Act: -0.0000

Data Index: 510076 (The Pile)

Max Activating Token Index: 148

Text #3

Max Range: 7.1346. Min Range: -7.1346

Max Act: 6.0434. Min Act: -0.0000

Data Index: 1971353 (The Pile)

Max Activating Token Index: 249

Text #4

Max Range: 7.1346. Min Range: -7.1346

Max Act: 5.7680. Min Act: -0.0000

Data Index: 1916291 (The Pile)

Max Activating Token Index: 223

Text #5

Max Range: 7.1346. Min Range: -7.1346

Max Act: 5.7531. Min Act: -0.0000

Data Index: 65127 (The Pile)

Max Activating Token Index: 457

Text #6

Max Range: 7.1346. Min Range: -7.1346

Max Act: 5.5221. Min Act: -0.0000

Data Index: 849328 (The Pile)

Max Activating Token Index: 785

Text #7

Max Range: 7.1346. Min Range: -7.1346

Max Act: 5.8285. Min Act: -0.0000

Data Index: 1979980 (The Pile)

Max Activating Token Index: 861

Text #8

Max Range: 7.1346. Min Range: -7.1346

Max Act: 5.2554. Min Act: -0.0000

Data Index: 1769976 (The Pile)

Max Activating Token Index: 486

Text #9

Max Range: 7.1346. Min Range: -7.1346

Max Act: 4.6499. Min Act: -0.0000

Data Index: 1181969 (The Pile)

Max Activating Token Index: 671

Text #10

Max Range: 7.1346. Min Range: -7.1346

Max Act: 4.7018. Min Act: -0.0000

Data Index: 117130 (The Pile)

Max Activating Token Index: 641

Text #11

Max Range: 7.1346. Min Range: -7.1346

Max Act: 4.6707. Min Act: 0.0000

Data Index: 1797794 (The Pile)

Max Activating Token Index: 64

Text #12

Max Range: 7.1346. Min Range: -7.1346

Max Act: 4.4277. Min Act: -0.0000

Data Index: 509470 (The Pile)

Max Activating Token Index: 720

Text #13

Max Range: 7.1346. Min Range: -7.1346

Max Act: 4.4831. Min Act: -0.0000

Data Index: 312220 (The Pile)

Max Activating Token Index: 784

Text #14

Max Range: 7.1346. Min Range: -7.1346

Max Act: 4.5958. Min Act: 0.0001

Data Index: 211286 (The Pile)

Max Activating Token Index: 35

Text #15

Max Range: 7.1346. Min Range: -7.1346

Max Act: 4.1050. Min Act: -0.0000

Data Index: 1083884 (The Pile)

Max Activating Token Index: 770

Text #16

Max Range: 7.1346. Min Range: -7.1346

Max Act: 4.3192. Min Act: -0.0000

Data Index: 703418 (The Pile)

Max Activating Token Index: 525

Text #17

Max Range: 7.1346. Min Range: -7.1346

Max Act: 4.4678. Min Act: -0.0000

Data Index: 572447 (The Pile)

Max Activating Token Index: 377

Text #18

Max Range: 7.1346. Min Range: -7.1346

Max Act: 4.2665. Min Act: -0.0000

Data Index: 1819445 (The Pile)

Max Activating Token Index: 273

Text #19

Max Range: 7.1346. Min Range: -7.1346

Max Act: 4.5957. Min Act: -0.0000

Data Index: 1088561 (The Pile)

Max Activating Token Index: 380