Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 303 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 6.5957. Min Range: -6.5957

Max Act: 6.5957. Min Act: -0.0000

Data Index: 1345140 (C4 (Web Text))

Max Activating Token Index: 101

Text #1

Max Range: 6.5957. Min Range: -6.5957

Max Act: 6.5547. Min Act: -0.0000

Data Index: 733992 (C4 (Web Text))

Max Activating Token Index: 953

Text #2

Max Range: 6.5957. Min Range: -6.5957

Max Act: 6.5599. Min Act: -0.0000

Data Index: 76450 (C4 (Web Text))

Max Activating Token Index: 32

Text #3

Max Range: 6.5957. Min Range: -6.5957

Max Act: 6.2482. Min Act: -0.0000

Data Index: 1251928 (C4 (Web Text))

Max Activating Token Index: 404

Text #4

Max Range: 6.5957. Min Range: -6.5957

Max Act: 6.1555. Min Act: -0.0000

Data Index: 977212 (C4 (Web Text))

Max Activating Token Index: 507

Text #5

Max Range: 6.5957. Min Range: -6.5957

Max Act: 6.0034. Min Act: -0.0000

Data Index: 1256588 (C4 (Web Text))

Max Activating Token Index: 536

Text #6

Max Range: 6.5957. Min Range: -6.5957

Max Act: 5.9943. Min Act: -0.0000

Data Index: 59956 (C4 (Web Text))

Max Activating Token Index: 392

Text #7

Max Range: 6.5957. Min Range: -6.5957

Max Act: 5.9688. Min Act: -0.0000

Data Index: 429066 (C4 (Web Text))

Max Activating Token Index: 63

Text #8

Max Range: 6.5957. Min Range: -6.5957

Max Act: 5.9155. Min Act: -0.0000

Data Index: 41138 (C4 (Web Text))

Max Activating Token Index: 74

Text #9

Max Range: 6.5957. Min Range: -6.5957

Max Act: 6.0034. Min Act: -0.0000

Data Index: 814715 (C4 (Web Text))

Max Activating Token Index: 126

Text #10

Max Range: 6.5957. Min Range: -6.5957

Max Act: 5.9065. Min Act: -0.0000

Data Index: 1012583 (C4 (Web Text))

Max Activating Token Index: 53

Text #11

Max Range: 6.5957. Min Range: -6.5957

Max Act: 5.7142. Min Act: -0.0000

Data Index: 803276 (C4 (Web Text))

Max Activating Token Index: 133

Text #12

Max Range: 6.5957. Min Range: -6.5957

Max Act: 5.1301. Min Act: -0.0000

Data Index: 231706 (C4 (Web Text))

Max Activating Token Index: 657

Text #13

Max Range: 6.5957. Min Range: -6.5957

Max Act: 5.8692. Min Act: -0.0000

Data Index: 1026070 (C4 (Web Text))

Max Activating Token Index: 108

Text #14

Max Range: 6.5957. Min Range: -6.5957

Max Act: 5.5032. Min Act: -0.0000

Data Index: 338780 (C4 (Web Text))

Max Activating Token Index: 240

Text #15

Max Range: 6.5957. Min Range: -6.5957

Max Act: 5.5078. Min Act: -0.0000

Data Index: 612167 (C4 (Web Text))

Max Activating Token Index: 632

Text #16

Max Range: 6.5957. Min Range: -6.5957

Max Act: 5.6579. Min Act: -0.0000

Data Index: 1048796 (C4 (Web Text))

Max Activating Token Index: 124

Text #17

Max Range: 6.5957. Min Range: -6.5957

Max Act: 5.8767. Min Act: -0.0000

Data Index: 442834 (C4 (Web Text))

Max Activating Token Index: 105

Text #18

Max Range: 6.5957. Min Range: -6.5957

Max Act: 5.0450. Min Act: -0.0000

Data Index: 539058 (C4 (Web Text))

Max Activating Token Index: 110

Text #19

Max Range: 6.5957. Min Range: -6.5957

Max Act: 5.3252. Min Act: -0.0000

Data Index: 1256647 (C4 (Web Text))

Max Activating Token Index: 1009