Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 633 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.5306. Min Act: -0.0000

Data Index: 251569 (The Pile)

Max Activating Token Index: 416

Text #1

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.4442. Min Act: -0.0000

Data Index: 101987 (The Pile)

Max Activating Token Index: 668

Text #2

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.4287. Min Act: -0.0000

Data Index: 1918690 (The Pile)

Max Activating Token Index: 41

Text #3

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.4435. Min Act: -0.0000

Data Index: 477430 (The Pile)

Max Activating Token Index: 206

Text #4

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.4006. Min Act: -0.0000

Data Index: 1487336 (The Pile)

Max Activating Token Index: 695

Text #5

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.3959. Min Act: -0.0000

Data Index: 233364 (The Pile)

Max Activating Token Index: 200

Text #6

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.4082. Min Act: -0.0000

Data Index: 1331940 (The Pile)

Max Activating Token Index: 105

Text #7

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.4243. Min Act: -0.0000

Data Index: 496171 (The Pile)

Max Activating Token Index: 146

Text #8

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.4009. Min Act: -0.0000

Data Index: 1459074 (The Pile)

Max Activating Token Index: 630

Text #9

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.4138. Min Act: -0.0000

Data Index: 723175 (The Pile)

Max Activating Token Index: 616

Text #10

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.3737. Min Act: -0.0000

Data Index: 1375282 (The Pile)

Max Activating Token Index: 105

Text #11

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.3642. Min Act: -0.0000

Data Index: 288951 (The Pile)

Max Activating Token Index: 433

Text #12

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.3766. Min Act: -0.0000

Data Index: 643122 (The Pile)

Max Activating Token Index: 326

Text #13

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.3527. Min Act: -0.0000

Data Index: 88867 (The Pile)

Max Activating Token Index: 294

Text #14

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.3329. Min Act: -0.0000

Data Index: 1115827 (The Pile)

Max Activating Token Index: 128

Text #15

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.3327. Min Act: -0.0000

Data Index: 1160572 (The Pile)

Max Activating Token Index: 104

Text #16

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.3512. Min Act: -0.0000

Data Index: 1449818 (The Pile)

Max Activating Token Index: 123

Text #17

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.3562. Min Act: -0.0000

Data Index: 1106829 (The Pile)

Max Activating Token Index: 919

Text #18

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.3550. Min Act: -0.0000

Data Index: 1951788 (The Pile)

Max Activating Token Index: 1013

Text #19

Max Range: 0.5306. Min Range: -0.5306

Max Act: 0.2681. Min Act: -0.0000

Data Index: 483682 (The Pile)

Max Activating Token Index: 931