Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4322 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.4912. Min Act: -0.0001

Data Index: 1647330 (Python Code)

Max Activating Token Index: 852

Text #1

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.4994. Min Act: -0.0001

Data Index: 1525728 (Python Code)

Max Activating Token Index: 625

Text #2

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.4602. Min Act: -0.0001

Data Index: 1363851 (Python Code)

Max Activating Token Index: 284

Text #3

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.4250. Min Act: -0.0001

Data Index: 1499909 (Python Code)

Max Activating Token Index: 83

Text #4

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.3248. Min Act: -0.0001

Data Index: 1573347 (Python Code)

Max Activating Token Index: 862

Text #5

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.3099. Min Act: -0.0001

Data Index: 1634246 (Python Code)

Max Activating Token Index: 561

Text #6

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.1983. Min Act: -0.0001

Data Index: 1427700 (Python Code)

Max Activating Token Index: 283

Text #7

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.3567. Min Act: -0.0001

Data Index: 1374732 (Python Code)

Max Activating Token Index: 394

Text #8

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.1692. Min Act: -0.0001

Data Index: 1365863 (Python Code)

Max Activating Token Index: 197

Text #9

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.1433. Min Act: -0.0001

Data Index: 1610331 (Python Code)

Max Activating Token Index: 57

Text #10

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.0988. Min Act: -0.0001

Data Index: 1588016 (Python Code)

Max Activating Token Index: 268

Text #11

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.1518. Min Act: -0.0001

Data Index: 1430455 (Python Code)

Max Activating Token Index: 483

Text #12

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.0653. Min Act: -0.0001

Data Index: 1523468 (Python Code)

Max Activating Token Index: 153

Text #13

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.0602. Min Act: -0.0001

Data Index: 1368419 (Python Code)

Max Activating Token Index: 635

Text #14

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.0189. Min Act: -0.0001

Data Index: 1481151 (Python Code)

Max Activating Token Index: 307

Text #15

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.1450. Min Act: -0.0001

Data Index: 1424362 (Python Code)

Max Activating Token Index: 224

Text #16

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.0439. Min Act: -0.0001

Data Index: 1584295 (Python Code)

Max Activating Token Index: 163

Text #17

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.0297. Min Act: -0.0001

Data Index: 1468492 (Python Code)

Max Activating Token Index: 571

Text #18

Max Range: 1.4994. Min Range: -1.4994

Max Act: 1.0611. Min Act: -0.0001

Data Index: 1435678 (Python Code)

Max Activating Token Index: 186

Text #19

Max Range: 1.4994. Min Range: -1.4994

Max Act: 0.8934. Min Act: -0.0001

Data Index: 1466942 (Python Code)

Max Activating Token Index: 401