Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3358 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.5932. Min Act: -0.0000

Data Index: 1586749 (Python Code)

Max Activating Token Index: 234

Text #1

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.6664. Min Act: -0.0000

Data Index: 1522942 (Python Code)

Max Activating Token Index: 551

Text #2

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.6118. Min Act: -0.0000

Data Index: 1406169 (Python Code)

Max Activating Token Index: 323

Text #3

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.6377. Min Act: -0.0000

Data Index: 1589945 (Python Code)

Max Activating Token Index: 180

Text #4

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.5940. Min Act: -0.0000

Data Index: 1562790 (Python Code)

Max Activating Token Index: 197

Text #5

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.5673. Min Act: -0.0000

Data Index: 1586927 (Python Code)

Max Activating Token Index: 900

Text #6

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.5777. Min Act: -0.0000

Data Index: 1409882 (Python Code)

Max Activating Token Index: 311

Text #7

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.6095. Min Act: -0.0000

Data Index: 1609578 (Python Code)

Max Activating Token Index: 920

Text #8

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.5741. Min Act: -0.0000

Data Index: 1474938 (Python Code)

Max Activating Token Index: 896

Text #9

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.5513. Min Act: -0.0000

Data Index: 1585317 (Python Code)

Max Activating Token Index: 192

Text #10

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.5491. Min Act: -0.0000

Data Index: 1656179 (Python Code)

Max Activating Token Index: 1009

Text #11

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.5344. Min Act: -0.0000

Data Index: 1411815 (Python Code)

Max Activating Token Index: 744

Text #12

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.5287. Min Act: -0.0000

Data Index: 1432877 (Python Code)

Max Activating Token Index: 856

Text #13

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.5730. Min Act: -0.0000

Data Index: 1395370 (Python Code)

Max Activating Token Index: 876

Text #14

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.5329. Min Act: -0.0000

Data Index: 1535118 (Python Code)

Max Activating Token Index: 555

Text #15

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.4922. Min Act: -0.0000

Data Index: 1396951 (Python Code)

Max Activating Token Index: 322

Text #16

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.4653. Min Act: -0.0000

Data Index: 1404184 (Python Code)

Max Activating Token Index: 166

Text #17

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.4947. Min Act: -0.0000

Data Index: 1581921 (Python Code)

Max Activating Token Index: 739

Text #18

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.5196. Min Act: -0.0000

Data Index: 1482373 (Python Code)

Max Activating Token Index: 976

Text #19

Max Range: 0.6664. Min Range: -0.6664

Max Act: 0.4668. Min Act: -0.0000

Data Index: 1523379 (Python Code)

Max Activating Token Index: 931