Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 328 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.4467. Min Act: 0.0000

Data Index: 1572578 (Python Code)

Max Activating Token Index: 235

Text #1

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.3481. Min Act: -0.0000

Data Index: 1568084 (Python Code)

Max Activating Token Index: 425

Text #2

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.3264. Min Act: -0.0000

Data Index: 1578730 (Python Code)

Max Activating Token Index: 92

Text #3

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.3227. Min Act: -0.0000

Data Index: 1406619 (Python Code)

Max Activating Token Index: 88

Text #4

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.3179. Min Act: -0.0001

Data Index: 1643775 (Python Code)

Max Activating Token Index: 989

Text #5

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.3117. Min Act: -0.0000

Data Index: 1626871 (Python Code)

Max Activating Token Index: 660

Text #6

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.3100. Min Act: -0.0000

Data Index: 1567380 (Python Code)

Max Activating Token Index: 176

Text #7

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.2874. Min Act: -0.0001

Data Index: 1636143 (Python Code)

Max Activating Token Index: 560

Text #8

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.2970. Min Act: -0.0000

Data Index: 1552232 (Python Code)

Max Activating Token Index: 261

Text #9

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.2987. Min Act: -0.0001

Data Index: 1614841 (Python Code)

Max Activating Token Index: 655

Text #10

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.2863. Min Act: -0.0001

Data Index: 1433768 (Python Code)

Max Activating Token Index: 643

Text #11

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.2784. Min Act: -0.0001

Data Index: 1591535 (Python Code)

Max Activating Token Index: 185

Text #12

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.2832. Min Act: -0.0000

Data Index: 1641433 (Python Code)

Max Activating Token Index: 364

Text #13

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.2502. Min Act: -0.0000

Data Index: 1387694 (Python Code)

Max Activating Token Index: 74

Text #14

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.2553. Min Act: -0.0001

Data Index: 1450488 (Python Code)

Max Activating Token Index: 200

Text #15

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.2650. Min Act: 0.0000

Data Index: 1395791 (Python Code)

Max Activating Token Index: 568

Text #16

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.2733. Min Act: -0.0000

Data Index: 1596154 (Python Code)

Max Activating Token Index: 194

Text #17

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.2608. Min Act: -0.0001

Data Index: 1638263 (Python Code)

Max Activating Token Index: 380

Text #18

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.2610. Min Act: -0.0001

Data Index: 1495545 (Python Code)

Max Activating Token Index: 668

Text #19

Max Range: 0.4467. Min Range: -0.4467

Max Act: 0.2495. Min Act: -0.0001

Data Index: 1615958 (Python Code)

Max Activating Token Index: 705