Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4997 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.5104. Min Act: -0.0000

Data Index: 472429 (C4 (Web Text))

Max Activating Token Index: 21

Text #1

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.3609. Min Act: -0.0000

Data Index: 1111230 (C4 (Web Text))

Max Activating Token Index: 816

Text #2

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.2803. Min Act: 0.0000

Data Index: 710972 (C4 (Web Text))

Max Activating Token Index: 317

Text #3

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.3277. Min Act: -0.0000

Data Index: 545604 (C4 (Web Text))

Max Activating Token Index: 284

Text #4

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.1977. Min Act: -0.0000

Data Index: 1003539 (C4 (Web Text))

Max Activating Token Index: 893

Text #5

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.1763. Min Act: -0.0000

Data Index: 24106 (C4 (Web Text))

Max Activating Token Index: 35

Text #6

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.2069. Min Act: 0.0000

Data Index: 422936 (C4 (Web Text))

Max Activating Token Index: 82

Text #7

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.1901. Min Act: -0.0000

Data Index: 1249529 (C4 (Web Text))

Max Activating Token Index: 908

Text #8

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.1691. Min Act: -0.0000

Data Index: 409019 (C4 (Web Text))

Max Activating Token Index: 31

Text #9

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.1688. Min Act: -0.0000

Data Index: 721627 (C4 (Web Text))

Max Activating Token Index: 236

Text #10

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.1639. Min Act: -0.0000

Data Index: 158697 (C4 (Web Text))

Max Activating Token Index: 443

Text #11

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.1538. Min Act: -0.0000

Data Index: 352021 (C4 (Web Text))

Max Activating Token Index: 272

Text #12

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.1316. Min Act: 0.0000

Data Index: 1150735 (C4 (Web Text))

Max Activating Token Index: 236

Text #13

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.0926. Min Act: -0.0000

Data Index: 594911 (C4 (Web Text))

Max Activating Token Index: 437

Text #14

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.0261. Min Act: 0.0000

Data Index: 463192 (C4 (Web Text))

Max Activating Token Index: 589

Text #15

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.0834. Min Act: -0.0000

Data Index: 825577 (C4 (Web Text))

Max Activating Token Index: 459

Text #16

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.0155. Min Act: -0.0000

Data Index: 147156 (C4 (Web Text))

Max Activating Token Index: 191

Text #17

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.0824. Min Act: -0.0000

Data Index: 487115 (C4 (Web Text))

Max Activating Token Index: 587

Text #18

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.0628. Min Act: -0.0000

Data Index: 1258653 (C4 (Web Text))

Max Activating Token Index: 448

Text #19

Max Range: 1.5104. Min Range: -1.5104

Max Act: 1.0606. Min Act: -0.0000

Data Index: 346619 (C4 (Web Text))

Max Activating Token Index: 609