Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 192 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1417. Min Range: -1.1417

Max Act: 1.1417. Min Act: -0.0000

Data Index: 1371635 (Python Code)

Max Activating Token Index: 733

Text #1

Max Range: 1.1417. Min Range: -1.1417

Max Act: 1.1002. Min Act: -0.0000

Data Index: 1373929 (Python Code)

Max Activating Token Index: 1002

Text #2

Max Range: 1.1417. Min Range: -1.1417

Max Act: 1.0391. Min Act: -0.0000

Data Index: 1052566 (C4 (Web Text))

Max Activating Token Index: 527

Text #3

Max Range: 1.1417. Min Range: -1.1417

Max Act: 0.9972. Min Act: -0.0000

Data Index: 1531394 (Python Code)

Max Activating Token Index: 729

Text #4

Max Range: 1.1417. Min Range: -1.1417

Max Act: 1.0581. Min Act: -0.0000

Data Index: 562606 (C4 (Web Text))

Max Activating Token Index: 123

Text #5

Max Range: 1.1417. Min Range: -1.1417

Max Act: 1.0311. Min Act: -0.0000

Data Index: 402287 (C4 (Web Text))

Max Activating Token Index: 798

Text #6

Max Range: 1.1417. Min Range: -1.1417

Max Act: 0.9437. Min Act: -0.0000

Data Index: 1621245 (Python Code)

Max Activating Token Index: 561

Text #7

Max Range: 1.1417. Min Range: -1.1417

Max Act: 0.8675. Min Act: -0.0000

Data Index: 1501167 (Python Code)

Max Activating Token Index: 68

Text #8

Max Range: 1.1417. Min Range: -1.1417

Max Act: 1.0957. Min Act: -0.0000

Data Index: 1543229 (Python Code)

Max Activating Token Index: 854

Text #9

Max Range: 1.1417. Min Range: -1.1417

Max Act: 0.9692. Min Act: -0.0000

Data Index: 1436574 (Python Code)

Max Activating Token Index: 329

Text #10

Max Range: 1.1417. Min Range: -1.1417

Max Act: 0.9043. Min Act: -0.0000

Data Index: 971439 (C4 (Web Text))

Max Activating Token Index: 830

Text #11

Max Range: 1.1417. Min Range: -1.1417

Max Act: 0.9445. Min Act: -0.0000

Data Index: 339915 (C4 (Web Text))

Max Activating Token Index: 139

Text #12

Max Range: 1.1417. Min Range: -1.1417

Max Act: 0.9549. Min Act: -0.0000

Data Index: 1387601 (Python Code)

Max Activating Token Index: 270

Text #13

Max Range: 1.1417. Min Range: -1.1417

Max Act: 1.0187. Min Act: -0.0000

Data Index: 950696 (C4 (Web Text))

Max Activating Token Index: 267

Text #14

Max Range: 1.1417. Min Range: -1.1417

Max Act: 0.9234. Min Act: -0.0000

Data Index: 1577382 (Python Code)

Max Activating Token Index: 183

Text #15

Max Range: 1.1417. Min Range: -1.1417

Max Act: 0.9466. Min Act: -0.0000

Data Index: 329554 (C4 (Web Text))

Max Activating Token Index: 65

Text #16

Max Range: 1.1417. Min Range: -1.1417

Max Act: 0.9035. Min Act: -0.0000

Data Index: 1238897 (C4 (Web Text))

Max Activating Token Index: 908

Text #17

Max Range: 1.1417. Min Range: -1.1417

Max Act: 0.8985. Min Act: -0.0000

Data Index: 776458 (C4 (Web Text))

Max Activating Token Index: 346

Text #18

Max Range: 1.1417. Min Range: -1.1417

Max Act: 0.9447. Min Act: -0.0000

Data Index: 327692 (C4 (Web Text))

Max Activating Token Index: 116

Text #19

Max Range: 1.1417. Min Range: -1.1417

Max Act: 0.9064. Min Act: -0.0000

Data Index: 1475370 (Python Code)

Max Activating Token Index: 732